Posts tagged Databases
The following two presentations are really useful for people just getting started with hosting websites on the Amazon AWS infrastructure.
These presentations were from guest speakers, the take home points for me were that Eagle Genomics have security tested AWS hosting and believe it to be secure enough for the kinds of sensitive data they are dealing with and that Servicetick were able to make good use of the data processing facilities
Another really useful item from Oleg Burlaca. AreoSQL is a free web based database management solution which, given some development could be a decent replacement for the awful phpMyAdmin. I say awful, phpMyAdmin is in a lot of ways actually a pretty good tool, however it's also notorious for going wrong when you least expect it and, to be honest needs a throrough interface overhaul. AreoSQL doesn't provide anything like the level of functionality that phpMyAdmin does (yet) BUT the interface is better, especially for people who like to use a componentised view of databases as a lot desktop app's provide. Overall I personally don't think it's going to beat other projects like Chive ( chive-project.com ) to being an immediate successor to phpMyAdmin but it's certainly an alternative to be considered...
More here: AeroSQL an alternative to phpMyAdmin
I can across a few other funky things in MySQL 5.1 today that I thought it might be worth telling you about in case you ever come across them too.
This time I was partitioning a number of large tables in and initially started to get the same weird errors as I did before when stupid queries were running away with themselves due to lack of temp space.
When you partition a table, MySQL seems to build a partitioned copy of it on the file system before swapping to that table for general use â€“ which seems like a fair way to go, but if you donâ€™t have enough temp space for the new table to be built in you get issues similar to those I discussed here, and you can get round them in the same way.
In my case, when doing this kind of maintenance I now add an â€˜overflowâ€™ path to the tmpdir variable which is basically a dir on part of the local filesystem with a large chunk of free space on it, but that isnâ€™t on the same partition as the MySQL tables themselves.
This allows these operations to spill over when they need to without causing a lot of hassle â€“ be warned though, itâ€™s not generally a good idea to use a fileshare on a NAS for this procedure! I donâ€™t know how well itâ€™d work with a SAN as I donâ€™t have one to play with, but doing it on a NAS will be REALLY slow in most cases and may cause other issues as a result.
But anyway thatâ€™s not the main thing I wanted to talk about.
What I wanted to talk about here is all your databases suddenly apparently vanishing altogether from MySQL after youâ€™ve implemented some a partitioning scheme. I guess the same would apply if you suddenly added a load of new databases/database tables to your MySQL setup too.
At the same time as vanishing databases Iâ€™ve also seen errors where MySQL reports that it cannot open the directory on the file system that a particular database resides in, and other similar filesystem related error messages.
What causes this to happen? Well it appears to be the a combination of the max number of connections, the max files mysql can open and the table cache, which dictates the number of files MySQL can have open at any one time.
Basically what seems to happen is that when table cache gets full, MySQL essentially dies not being able to open any more database files, and therefore cannot access any more information either. BUT because the main operation on the server is not interrupted it seems that the MySQL process doesnâ€™t die - it just continues to run but without access to any information, as if none of it ever existed!
This tends to happen after partitioning has been done because a partitioning scheme can result in a large number of new database data files being produced â€“ after all, thatâ€™s all itâ€™s doing, breaking one massive single myISAM file down into more manageable chunks.
Also another reason it tends to happen after partitioning is because partitioning came in in MySQL 5.1 and in MySQL 5.1 the name of the table cache server variable (at least) changed from
table_open_cache. So, if your my.cnf or my.ini still references table_cache and your running 5.1.3 upwards youâ€™ll not actually be setting this value any more and as a result the server will revert to its default value â€“ namely 64 â€“ a tiny amount.
I have seen recommendations that you should set this value to around to 2048 for a lot of systems (which seems a bit arbitrary really), but the way to determine what kind of numbers you should be using here comes from analysing the results of the opened_files server status variable traded off against the max connections, and whatever the table cache is currently set to - see the resources links below for more info on this â€“ you can access this status variable and a few other useful related parameters by running:
SHOW STATUS LIKE '%open%';
Issues can also ensue when the number of files opened by the server exceeds/hits the limit of the total number of files the operating system allows any one user to open, and if this actually appears to be the case you need to check the docs for your OS in order to up this limit.
To get an idea of the maximum number of tables mysql might look to open at any one time you can count the number of files in the mysql data dir - this dir might also contain other junk like bin logs, error logs, etc. but for a rough upper limit this will do it:
ls -1R | wc -l
If your data dir is polluted with logs then you can filter this kind of a count through grep and add up the results:
ls -1R | grep *.MYI | wc -l
ls -1R | grep *.MYD | wc -l
To find out the number of files mysql has open currently you can also do this:
$shell> lsof | grep mysql | wc â€“l
You can find out the number of max open files your OS supports by running this:
$shell> cat /proc/sys/fs/file-max
Some resources on these issues: