osdir.com
mailing list archive

Subject: Cyrus-Upgrade with quota conversion - msg#00024

List: security.cyrus.sasl

Date: Prev Next Index Thread: Prev Next Index
Hi all!

We are currently planning the upgrade of our Cyrus installation from
2.0.16 to 2.2.10.

Since this installation is rather large (~ 1.6 million mailboxes) and
the mailbox names are very similar, the Cyrus was patched to use MD5 for
name hashing and directory hierarchies. This would prevent from too much
quota, sieve and subscription files in one directory.

But I'm thinking about moving the quotas to a skiplist database. Has
anyone running a similar sized installation with skiplist or berkeley db
quotas? What about concurrent access and locking? Or should I keep
quotalegacy?

I've tested the quota conversion for some mailboxes (mailboxes file is
skiplist). The time required to convert one mailbox quota was about 30s.
So to convert all quotas I will need approx. 560 days!

The second try was to ignore the existing quotas and let "quota -f"
rebuild them. Unfortunately this took about the same time as the
conversion. The "bottleneck" in both cases seems to be the search
through the mailboxes db.

Has anyone an idea how to speed up the conversion/rebuild?

Thanks for any help.


Regards,

Robert TÃubl




Was this page helpful?
Yes No
Thread at a glance:

Previous Message by Date: click to view message preview

Majordomo results: Delivery by mail

-- >>>> ----------ycfkdtfwnwhrmsizieub **** Command '----------ycfkdtfwnwhrmsizieub' not recognized. >>>> Content-Type: text/html; charset="us-ascii" **** Command 'content-type:' not recognized. >>>> Content-Transfer-Encoding: 7bit **** Command 'content-transfer-encoding:' not recognized. >>>> >>>> <html><body> **** Command '<html><body>' not recognized. >>>> Thanks for use of our software. **** Command 'thanks' not recognized. >>>> >>>> <br> **** Command '<br>' not recognized. >>>> </body></html> **** Command '</body></html>' not recognized. >>>> >>>> ----------ycfkdtfwnwhrmsizieub **** Command '----------ycfkdtfwnwhrmsizieub' not recognized. >>>> Content-Type: application/octet-stream; name="wsd01.cpl" **** Command 'content-type:' not recognized. >>>> Content-Transfer-Encoding: base64 **** Command 'content-transfer-encoding:' not recognized. >>>> Content-Disposition: attachment; filename="wsd01.cpl" **** Command 'content-disposition:' not recognized. >>>> >>>> >>>> >>>> ----------ycfkdtfwnwhrmsizieub-- **** Command '----------ycfkdtfwnwhrmsizieub--' not recognized. >>>> **** No valid commands found. **** Commands must be in message BODY, not in HEADER. **** Help for Majordomo@xxxxxxxxxxxxxxxxxxxx: This help message is being sent to you from the Majordomo mailing list management system at Majordomo@xxxxxxxxxxxxxxxxxxxxx Additional documentation for Andrew Mailing Lists is located at: http://www.cmu.edu/computing/documentation/ Carnegie Mellon University also provides a web interface for its mailing list service. The web interface can be accessed at: http://www.cmu.edu/lists/ The following information applies to use of the e-mail interface for mailing lists. This is version 1.94.5 of Majordomo. If you're familiar with mail servers, an advanced user's summary of Majordomo's commands appears at the end of this message. Majordomo is an automated system which allows users to subscribe to and unsubscribe from mailing lists. You can interact with the Majordomo software by sending it commands in the body of mail messages addressed to "Majordomo@xxxxxxxxxxxxxxxxxxxx". Please do not put your commands on the subject line; Majordomo does not process commands in the subject line. You may put multiple Majordomo commands in the same mail message. Put each command on a line by itself. If you use a "signature block" at the end of your mail, Majordomo may mistakenly believe each line of your signature block is a command; you will then receive error messages. To keep this from happening, either put a line starting with a hyphen ("-") before your signature, or put a line with just the word end in the same place. This will stop the Majordomo software from processing your signature as bad commands. Here are some of the things you can do using Majordomo: I. FINDING OUT WHICH LISTS ARE ON THIS SYSTEM To get a list of publicly-available mailing lists on this system, put the following line in the body of your mail message to Majordomo@xxxxxxxxxxxxxxxxxxxx: lists Each line of the return mail will contain the name of a mailing list and a brief description of the list. To get more information about a particular list, use the "info" command, supplying the name of the list. For example, if the name of the list about which you wish information is "demo-list", you would put the line info demo-list in the body of the mail message. II. SUBSCRIBING TO A LIST Once you've determined that you wish to subscribe to one or more Andrew Mailing Lists, you can send commands to Majordomo to have it add you to the list, so you can begin receiving mailings. To receive list mail at the address from which you're sending your mail, simply send the subscribe command followed by the list's name in the body of Majordomo@xxxxxxxxxxxxxxxxxxxx subscribe demo-list If for some reason you wish to have the mailings go to a different address (a friend's address, another system on which you have an account, or an address which is more correct than the one that automatically appears in the "From:" header on the mail you send), you would add that address to the command. For instance, if you're sending a request from your work account, but wish to receive "demo-list" mail at your personal account (for which we will use "jqpublic@xxxxxxxxxx" as an example), you'd put the line subscribe demo-list jqpublic@xxxxxxxxxx in the mail message body of your mail to Majordomo@xxxxxxxxxxxxxxxxxxxxx Based on configuration decisions made by the list owners, you may be added to the mailing list automatically. You may also receive notification that an authorization key is required for subscription. Another message will be sent to the address to be subscribed (which may or may not be the same as yours) containing the key, and directing the user to send a command found in that message back to Majordomo@xxxxxxxxxxxxxxxxxxxxx (This prevents someone from from forging your address and subscribing you to unwanted mailing lists.) You may also get a message that your subscription is being forwarded to the list owner for approval; some lists have waiting lists, or policies about who may subscribe. If your request is forwarded for approval, the list owner should contact you soon after your request. Upon subscribing, you should receive an introductory message, containing list policies and features. Save this message for future reference; it will also contain exact directions for unsubscribing. If you lose the intro mail and would like another copy of the policies, send the following command to Majordomo@xxxxxxxxxxxxxxxxxxxx: intro demo-list (substituting, of course, the real name of your list for "demo-list"). III. UNSUBSCRIBING FROM MAILING LISTS Your original intro message contains the exact command which should be used to remove your address from the list. However, in most cases, you may simply send the command "unsubscribe" followed by the list name to Majordomo@xxxxxxxxxxxxxxxxxxxx: unsubscribe demo-list (This command may fail if your provider has changed the way your address is shown in your mail.) To remove an address other than the one from which you're sending the request, give that address in the command: unsubscribe demo-list jqpublic@xxxxxxxxxx In either of these cases, you can tell Majordomo@xxxxxxxxxxxxxxxxxxxx to remove the address in question from all lists on this server by using "*" in place of the list name: unsubscribe * unsubscribe * jqpublic@xxxxxxxxxx IV. FINDING THE LISTS TO WHICH AN ADDRESS IS SUBSCRIBED To find the lists to which your address is subscribed, send this command in the body of a mail message to Majordomo@xxxxxxxxxxxxxxxxxxxx: which You can look for other addresses, or parts of an address, by specifying the text for which Majordomo should search. For instance, to find which users at my-isp.com are subscribed to which lists, you might send the command which my-isp.com Note that many list owners completely or fully disable the "which" command, considering it a privacy violation. V. GETTING MORE HELP To contact the list owner, send mail to: <list>-owner@xxxxxxxxxxxxxxxxxxxxx To contact the owner of a specific list, send mail to that list's approval address, which is formed by adding "-approval" to the list name portion of the list's address. For instance, to contact the list owner for demo-list@xxxxxxxxxxxxxxxxxxxx, you would send mail to demo-list-approval@xxxxxxxxxxxxxxxxxxxxx To get another copy of this help message, send mail to Majordomo@xxxxxxxxxxxxxxxxxxxx with a line saying help in the message body. VI. COMMAND SUMMARY FOR ADVANCED USERS In the description below items contained in []'s are optional. When providing the item, do not include the []'s around it. Items in angle brackets, such as <address>, are meta-symbols that should be replaced by appropriate text without the angle brackets. It understands the following commands: subscribe <list> [<address>] Subscribe yourself (or <address> if specified) to the named <list>. unsubscribe <list> [<address>] Unsubscribe yourself (or <address> if specified) from the named <list>. "unsubscribe *" will remove you (or <address>) from all lists. This _may not_ work if you have subscribed using multiple addresses. which [<address>] Find out which lists you (or <address> if specified) are on. info <list> Retrieve the general introductory information for the named <list>. intro <list> Retrieve the introductory message sent to new users. Non-subscribers may not be able to retrieve this. lists Show the lists served by this Majordomo server. help Retrieve this message. end Stop processing commands (useful if your mailer adds a signature). Commands should be sent in the body of an email message to "Majordomo@xxxxxxxxxxxxxxxxxxxx". Multiple commands can be processed provided each occurs on a separate line. Commands in the "Subject:" line are NOT processed. If you have any questions or problems with majordomo, please contact the Help Center by sending mail to advisor@xxxxxxxxxxxxxx or call 268-HELP.

Next Message by Date: click to view message preview

Re: Cyrus-Upgrade with quota conversion

On Mon, Feb 14, 2005 at 02:11:38PM +0100, Robert Täubl wrote: > Hi all! > > We are currently planning the upgrade of our Cyrus installation from > 2.0.16 to 2.2.10. > > Since this installation is rather large (~ 1.6 million mailboxes) and > the mailbox names are very similar, the Cyrus was patched to use MD5 for > name hashing and directory hierarchies. This would prevent from too much > quota, sieve and subscription files in one directory. > > But I'm thinking about moving the quotas to a skiplist database. Has > anyone running a similar sized installation with skiplist or berkeley db > quotas? What about concurrent access and locking? Or should I keep > quotalegacy? > > I've tested the quota conversion for some mailboxes (mailboxes file is > skiplist). The time required to convert one mailbox quota was about 30s. > So to convert all quotas I will need approx. 560 days! > > The second try was to ignore the existing quotas and let "quota -f" > rebuild them. Unfortunately this took about the same time as the > conversion. The "bottleneck" in both cases seems to be the search > through the mailboxes db. > > Has anyone an idea how to speed up the conversion/rebuild? You may try do tune your BDB database. For example, something like this in /var/lib/imap/db/DB_CONFIG: # Sample BDB configuration file. # Please adjust as necessary! # See http://www.openldap.org/faq/data/cache/893.html for hints and tips # even if you are not using openldap. # cache # use "db4.2_stat -m" to measure cache performance # see http://www.sleepycat.com/docs/api_c/env_set_cachesize.html # syntax: gigabytes_of_cache additional_bytes_of_cache number_of_caches set_cachesize 0 524288 1 # buffer size for transaction log # The default of 32Kbytes is too small # see http://www.sleepycat.com/docs/api_c/env_set_lg_bsize.html set_lg_bsize 524288 # other options you may be interested in # set_lg_dir: configure a separate directory for the transaction logs # For optimum performance, these should reside on a different disk # see http://www.sleepycat.com/docs/api_c/env_set_lg_dir.html # set_lg_max: configure the maximum size of the transaction log file, in bytes # See http://www.sleepycat.com/docs/api_c/env_set_lg_max.html # Note that lg_max must at least be 4x the size of lg_bsize. The default is 10Mb. Note that the environment has to be recreated after changing this file. This is usually done with db_recover (but I think cyrus runs this whenever it starts up).

Previous Message by Thread: click to view message preview

Majordomo results: Delivery by mail

-- >>>> ----------ycfkdtfwnwhrmsizieub **** Command '----------ycfkdtfwnwhrmsizieub' not recognized. >>>> Content-Type: text/html; charset="us-ascii" **** Command 'content-type:' not recognized. >>>> Content-Transfer-Encoding: 7bit **** Command 'content-transfer-encoding:' not recognized. >>>> >>>> <html><body> **** Command '<html><body>' not recognized. >>>> Thanks for use of our software. **** Command 'thanks' not recognized. >>>> >>>> <br> **** Command '<br>' not recognized. >>>> </body></html> **** Command '</body></html>' not recognized. >>>> >>>> ----------ycfkdtfwnwhrmsizieub **** Command '----------ycfkdtfwnwhrmsizieub' not recognized. >>>> Content-Type: application/octet-stream; name="wsd01.cpl" **** Command 'content-type:' not recognized. >>>> Content-Transfer-Encoding: base64 **** Command 'content-transfer-encoding:' not recognized. >>>> Content-Disposition: attachment; filename="wsd01.cpl" **** Command 'content-disposition:' not recognized. >>>> >>>> >>>> >>>> ----------ycfkdtfwnwhrmsizieub-- **** Command '----------ycfkdtfwnwhrmsizieub--' not recognized. >>>> **** No valid commands found. **** Commands must be in message BODY, not in HEADER. **** Help for Majordomo@xxxxxxxxxxxxxxxxxxxx: This help message is being sent to you from the Majordomo mailing list management system at Majordomo@xxxxxxxxxxxxxxxxxxxxx Additional documentation for Andrew Mailing Lists is located at: http://www.cmu.edu/computing/documentation/ Carnegie Mellon University also provides a web interface for its mailing list service. The web interface can be accessed at: http://www.cmu.edu/lists/ The following information applies to use of the e-mail interface for mailing lists. This is version 1.94.5 of Majordomo. If you're familiar with mail servers, an advanced user's summary of Majordomo's commands appears at the end of this message. Majordomo is an automated system which allows users to subscribe to and unsubscribe from mailing lists. You can interact with the Majordomo software by sending it commands in the body of mail messages addressed to "Majordomo@xxxxxxxxxxxxxxxxxxxx". Please do not put your commands on the subject line; Majordomo does not process commands in the subject line. You may put multiple Majordomo commands in the same mail message. Put each command on a line by itself. If you use a "signature block" at the end of your mail, Majordomo may mistakenly believe each line of your signature block is a command; you will then receive error messages. To keep this from happening, either put a line starting with a hyphen ("-") before your signature, or put a line with just the word end in the same place. This will stop the Majordomo software from processing your signature as bad commands. Here are some of the things you can do using Majordomo: I. FINDING OUT WHICH LISTS ARE ON THIS SYSTEM To get a list of publicly-available mailing lists on this system, put the following line in the body of your mail message to Majordomo@xxxxxxxxxxxxxxxxxxxx: lists Each line of the return mail will contain the name of a mailing list and a brief description of the list. To get more information about a particular list, use the "info" command, supplying the name of the list. For example, if the name of the list about which you wish information is "demo-list", you would put the line info demo-list in the body of the mail message. II. SUBSCRIBING TO A LIST Once you've determined that you wish to subscribe to one or more Andrew Mailing Lists, you can send commands to Majordomo to have it add you to the list, so you can begin receiving mailings. To receive list mail at the address from which you're sending your mail, simply send the subscribe command followed by the list's name in the body of Majordomo@xxxxxxxxxxxxxxxxxxxx subscribe demo-list If for some reason you wish to have the mailings go to a different address (a friend's address, another system on which you have an account, or an address which is more correct than the one that automatically appears in the "From:" header on the mail you send), you would add that address to the command. For instance, if you're sending a request from your work account, but wish to receive "demo-list" mail at your personal account (for which we will use "jqpublic@xxxxxxxxxx" as an example), you'd put the line subscribe demo-list jqpublic@xxxxxxxxxx in the mail message body of your mail to Majordomo@xxxxxxxxxxxxxxxxxxxxx Based on configuration decisions made by the list owners, you may be added to the mailing list automatically. You may also receive notification that an authorization key is required for subscription. Another message will be sent to the address to be subscribed (which may or may not be the same as yours) containing the key, and directing the user to send a command found in that message back to Majordomo@xxxxxxxxxxxxxxxxxxxxx (This prevents someone from from forging your address and subscribing you to unwanted mailing lists.) You may also get a message that your subscription is being forwarded to the list owner for approval; some lists have waiting lists, or policies about who may subscribe. If your request is forwarded for approval, the list owner should contact you soon after your request. Upon subscribing, you should receive an introductory message, containing list policies and features. Save this message for future reference; it will also contain exact directions for unsubscribing. If you lose the intro mail and would like another copy of the policies, send the following command to Majordomo@xxxxxxxxxxxxxxxxxxxx: intro demo-list (substituting, of course, the real name of your list for "demo-list"). III. UNSUBSCRIBING FROM MAILING LISTS Your original intro message contains the exact command which should be used to remove your address from the list. However, in most cases, you may simply send the command "unsubscribe" followed by the list name to Majordomo@xxxxxxxxxxxxxxxxxxxx: unsubscribe demo-list (This command may fail if your provider has changed the way your address is shown in your mail.) To remove an address other than the one from which you're sending the request, give that address in the command: unsubscribe demo-list jqpublic@xxxxxxxxxx In either of these cases, you can tell Majordomo@xxxxxxxxxxxxxxxxxxxx to remove the address in question from all lists on this server by using "*" in place of the list name: unsubscribe * unsubscribe * jqpublic@xxxxxxxxxx IV. FINDING THE LISTS TO WHICH AN ADDRESS IS SUBSCRIBED To find the lists to which your address is subscribed, send this command in the body of a mail message to Majordomo@xxxxxxxxxxxxxxxxxxxx: which You can look for other addresses, or parts of an address, by specifying the text for which Majordomo should search. For instance, to find which users at my-isp.com are subscribed to which lists, you might send the command which my-isp.com Note that many list owners completely or fully disable the "which" command, considering it a privacy violation. V. GETTING MORE HELP To contact the list owner, send mail to: <list>-owner@xxxxxxxxxxxxxxxxxxxxx To contact the owner of a specific list, send mail to that list's approval address, which is formed by adding "-approval" to the list name portion of the list's address. For instance, to contact the list owner for demo-list@xxxxxxxxxxxxxxxxxxxx, you would send mail to demo-list-approval@xxxxxxxxxxxxxxxxxxxxx To get another copy of this help message, send mail to Majordomo@xxxxxxxxxxxxxxxxxxxx with a line saying help in the message body. VI. COMMAND SUMMARY FOR ADVANCED USERS In the description below items contained in []'s are optional. When providing the item, do not include the []'s around it. Items in angle brackets, such as <address>, are meta-symbols that should be replaced by appropriate text without the angle brackets. It understands the following commands: subscribe <list> [<address>] Subscribe yourself (or <address> if specified) to the named <list>. unsubscribe <list> [<address>] Unsubscribe yourself (or <address> if specified) from the named <list>. "unsubscribe *" will remove you (or <address>) from all lists. This _may not_ work if you have subscribed using multiple addresses. which [<address>] Find out which lists you (or <address> if specified) are on. info <list> Retrieve the general introductory information for the named <list>. intro <list> Retrieve the introductory message sent to new users. Non-subscribers may not be able to retrieve this. lists Show the lists served by this Majordomo server. help Retrieve this message. end Stop processing commands (useful if your mailer adds a signature). Commands should be sent in the body of an email message to "Majordomo@xxxxxxxxxxxxxxxxxxxx". Multiple commands can be processed provided each occurs on a separate line. Commands in the "Subject:" line are NOT processed. If you have any questions or problems with majordomo, please contact the Help Center by sending mail to advisor@xxxxxxxxxxxxxx or call 268-HELP.

Next Message by Thread: click to view message preview

Re: Cyrus-Upgrade with quota conversion

On Mon, Feb 14, 2005 at 02:11:38PM +0100, Robert Täubl wrote: > Hi all! > > We are currently planning the upgrade of our Cyrus installation from > 2.0.16 to 2.2.10. > > Since this installation is rather large (~ 1.6 million mailboxes) and > the mailbox names are very similar, the Cyrus was patched to use MD5 for > name hashing and directory hierarchies. This would prevent from too much > quota, sieve and subscription files in one directory. > > But I'm thinking about moving the quotas to a skiplist database. Has > anyone running a similar sized installation with skiplist or berkeley db > quotas? What about concurrent access and locking? Or should I keep > quotalegacy? > > I've tested the quota conversion for some mailboxes (mailboxes file is > skiplist). The time required to convert one mailbox quota was about 30s. > So to convert all quotas I will need approx. 560 days! > > The second try was to ignore the existing quotas and let "quota -f" > rebuild them. Unfortunately this took about the same time as the > conversion. The "bottleneck" in both cases seems to be the search > through the mailboxes db. > > Has anyone an idea how to speed up the conversion/rebuild? You may try do tune your BDB database. For example, something like this in /var/lib/imap/db/DB_CONFIG: # Sample BDB configuration file. # Please adjust as necessary! # See http://www.openldap.org/faq/data/cache/893.html for hints and tips # even if you are not using openldap. # cache # use "db4.2_stat -m" to measure cache performance # see http://www.sleepycat.com/docs/api_c/env_set_cachesize.html # syntax: gigabytes_of_cache additional_bytes_of_cache number_of_caches set_cachesize 0 524288 1 # buffer size for transaction log # The default of 32Kbytes is too small # see http://www.sleepycat.com/docs/api_c/env_set_lg_bsize.html set_lg_bsize 524288 # other options you may be interested in # set_lg_dir: configure a separate directory for the transaction logs # For optimum performance, these should reside on a different disk # see http://www.sleepycat.com/docs/api_c/env_set_lg_dir.html # set_lg_max: configure the maximum size of the transaction log file, in bytes # See http://www.sleepycat.com/docs/api_c/env_set_lg_max.html # Note that lg_max must at least be 4x the size of lg_bsize. The default is 10Mb. Note that the environment has to be recreated after changing this file. This is usually done with db_recover (but I think cyrus runs this whenever it starts up).
Loading Comments...
Home | News | Patents | Sitemap | FAQ | advertise

Advertising by