Synopsis:
Insert queries executed from a POSIX thread cause a segment fault when the
query exceeds the size of about 70KB. When not using threads, the library
doesn't cause a segment fault until the query reaches a few megs (tested
reliably with a 32MB file, which I still consider to be a problem).
I isolated the failure from my main line of development and worked it into
this fairly minimal test program. My application is daemonized and
threaded, and it is critical that it be able to reliably make inserts of
files up to 20-30MB in size. I could probably live with 15MB, but
instability at 55KB and consistent crashes at 120KB inserts are completely
unworkable.
I've tested the program in many configurations with a number of data files
and produced three principal backtraces, which are outlined below.
I looked through the sql_query#.hh files and sql_query.cc, particularly at
the SQLQuery::str() functions, but I can't say I understand what's under the
hood of MySQL++ enough to make any guesses on what's giving me trouble.
I have really run out of ideas on how to make this work.
So my questions:
I'm not missing anything obvious, am I? ;)
Where's the problem?
What do I need to do in terms of changes or workarounds to make this work
stably?
I'm running mysql++-1.7.9 on a FreeBSD 4.9-STABLE box with MySQL version
4.0.16.
Let me know if you have any questions, and sorry about the heavy post ;)
Thanks in advance,
--Travis
// File: test.cpp
//
// Simple test program to isolate
// the segment fault that occurs
// when execute() or preview() is
// called on a large query.
//
// Compile with
// g++ test.cpp -o test -lsqlplus -lccgnu2 -pthread
-I/usr/local/include/mysql
//
// Given a test file test.data in the current directory, run with
// ./test test.data
//
// Failure conditions and gdb backtraces
/*
** Threading is used and query is assembled
** using a simple %0q template and .def[].
** When compiled and run with this configuration,
** the program exits with a segment fault
** (occasionally with an illegal instruction instead)
** The program with work with a small input file (<65KB)
** but with a 120KB file, it always fails on my system.
** The program will work without a hitch (until the files
** get really big) if threading is not used.
** I've also tried using char* arrays instead of strings
** throughout the program, but the result is the same.
** (gdb) bt
** #0 0x280a805c in SQLQuery::str () from /usr/local/lib/libsqlplus.so.1
** #1 0x8054e3f in MysqlQuery::preview ()
** #2 0x804e728 in DB::commit ()
** #3 0x804e153 in DB::run ()
** #4 0x280d2821 in ccxx_exec_handler () from
/usr/local/lib/libccgnu2-1.0.so.0
** #5 0x2816811c in _thread_start () from /usr/lib/libc_r.so.4
** #6 0x0 in ?? ()
**
** Same configuration as above, except that
** instead of using a templated query, we
** directly insert the data into the query
** with the insertion operator.
** Again, the program works when I remove
** the threading.
** With threading, the segment fault occurs
** when using a file only slightly larger than
** above. An input file of 120KB should
** still produce a reliable segment fault.
** (gdb) bt
** #0 0x281d8f92 in memcpy () from /usr/lib/libc_r.so.4
** #1 0x0 in ?? ()
**
** Here I reconfigured to not use threads.
** As mentioned above, it will work with
** a 120KB file, with some room above that,
** but when the files get big (32MB), it fails
** with a segment fault and this bt.
** (gdb) bt
** #0 0x280a505c in SQLQuery::str () from /usr/local/lib/libsqlplus.so.1
** #1 0x280a5111 in SQLQuery::str () from /usr/local/lib/libsqlplus.so.1
** #2 0x805382f in MysqlQuery::execute ()
** #3 0x8052e8d in MysqlQuery::execute ()
** #4 0x804d289 in DB::commit ()
** #5 0x804de1b in main ()
** #6 0x804cb7a in _start ()
*/
// For the data files, I used 64-bit encoded data,
// like found in email attachments.
// The problem areas are indicated in the code.
#include <iostream>
#include <fstream>
#include <string>
// Use GNU Common C++ Libraries
// http://www.gnu.org/software/commoncpp/
#include <cc++/socket.h>
#include <cc++/process.h>
// Use MySQL++ Libraries
#include <sqlplus.hh>
using namespace std;
using namespace ost;
// Create a class to hold the data to be inserted
// and to manage the thread.
class DB : public Thread
{
protected:
const string* Data;
Connection* myConnection;
void run();
void final()
{
delete this;
}
static Mutex mutex;
public:
DB();
~DB();
void setData(const string& data);
void commit();
};
// Use a static mutex to prevent concurrent database operations.
Mutex DB::mutex;
DB::DB()
{
myConnection = new Connection(use_exceptions);
string database;
string host;
string user;
string password;
database = "test_db";
host = "localhost";
user = "root";
password = "";
myConnection->connect(database.c_str(), host.c_str(), user.c_str(),
password.c_str());
}
DB::~DB()
{
delete Data;
}
void DB::run()
{
// This is wrapped around like this so I can unthread this
// test to prove a point.
commit();
}
void DB::setData(const string& data)
{
Data = &data;
}
void DB::commit()
{
try
{
mutex.enterMutex();
cout << "OK, ready to make a query object." << endl;
Query query = myConnection->query();
query << "insert into data (data) values ('";
query << (*Data) << "')";
// FYI: This sequence, which is preferable to me,
// fails under nearly the same conditions as the way
// I have it currently uncommented.
// Doing it this way causes a segfault at a slightly lower
// data size threshold when threaded.
// query << "insert into data (data) values (%0:data)";
// query.parse();
// query.def["data"] = *Data;
// FYI: query.preview() failes under the same conditions
// as query.execute()
// cout << "Here's the query we are ready to execute: " << endl;
// cout << query.preview() << endl;
cout << "Ready to execute the query" << endl;
// ************************************************************
// This next function causes the program to segfault
// (unless query.preview() is uncommented, in which case
// it segfaults instead)
// ************************************************************
query.execute();
cout << "Executed the query" << endl;
mutex.leaveMutex();
}
catch (BadQuery &er)
{
cout << "Query Error: " << er.error << endl;
}
catch (BadConversion er)
{
string data = er.data;
string typeName = er.type_name;
cout << "Error: " << data << " " << typeName << endl;
}
catch (exception &er)
{
cout << "Error: " << er.what() << endl;
}
}
string readStdin();
string readStdin()
{
string data;
char next;
while (!cin.eof())
{
next = cin.get();
data += next;
}
data.erase(data.size()-1,1);
return data;
}
// This is faster than standard input
string readFile(const string& filename);
string readFile(const string& filename)
{
int length;
char * buffer;
ifstream is;
is.open(filename.c_str(), ios::in);
is.seekg(0, ios::end);
length = is.tellg();
is.seekg(0, ios::beg);
buffer = new char[length];
is.read(buffer, length);
is.close();
return buffer;
}
int main(int argc, char** argv)
{
// Whether or not the main process
// is detached doesn't seem to affect
// the outcome.
// Process::detach();
string data;
if (argc < 2)
{
cout << "Reading Stdin" << endl;
data = readStdin();
}
else
{
cout << "Reading File" << endl;
data = readFile(argv[1]);
}
DB* myDB = new DB;
myDB->setData(data);
// Here's where it gets interesting.
// If I forget about threading for the moment
// and run the same code unthreaded (compiled the same way)
// by commenting out myDB->detach() and uncommenting myDB->commit()
// then everything works fine with the 120KB data files.
// myDB->commit();
// delete myDB;
myDB->detach();
wait(3000);
return 0;
}
/////////////////////////////////////////////////////////////
/*
Side Note:
The program compiles with the following warnings, which I
ignore as instructed in "Re: MySQL++ & pthread adverse reaction" (2/10/2004)
travis@dev% g++ test6.cpp -o test -lpme -lsqlplus -lccgnu2 -pthread
-I/usr/local/include/mysql
/usr/lib/libc.so.4: WARNING! setkey(3) not present in the system!
/usr/lib/libc.so.4: warning: this program uses gets(), which is unsafe.
/usr/lib/libc.so.4: warning: mktemp() possibly used unsafely; consider using
mkstemp()
/usr/lib/libc.so.4: WARNING! des_setkey(3) not present in the system!
/usr/lib/libc.so.4: WARNING! encrypt(3) not present in the system!
/usr/lib/libc.so.4: warning: tmpnam() possibly used unsafely; consider using
mkstemp()
/usr/lib/libc.so.4: warning: this program uses f_prealloc(), which is not
recommended.
/usr/lib/libc.so.4: WARNING! des_cipher(3) not present in the system!
/usr/lib/libc.so.4: warning: tempnam() possibly used unsafely; consider
using mkstemp()
*/
--
MySQL++ Mailing List
For list archives: http://lists.mysql.com/plusplus
To unsubscribe:
http://lists.mysql.com/plusplus?unsub=gcdmc-plusplus@xxxxxxxxxxx
|
