February 24, 2010 at 2:25 pm (PHP)
Tags: php upload file size, php.ini
PHP file upload max size is determined by 2 configuration values in php.ini, namely upload_max_filesize and post_max_size. You can get the maximum file size allowed in uploading by this snippet:
$max_upload = (int)(ini_get('upload_max_filesize'));
$max_post = (int)(ini_get('post_max_size'));
$upload_mb = $max_upload <= $max_post ? $max_upload : $max_post;
$upload_mb is the maximum file size allowed for upload in MB. Just display this value beside the file upload control so the user knows the limit.
Author: Yang Yang
February 6, 2010 at 6:30 am (mysql)
Tags: mysql, optimizing mysql
What one can and should optimize
- OS / libraries
- SQL server (setup and queries)
Optimizing hardware for MySQL
- If you need big tables ( > 2G), you should consider using 64 bit
hardware like Alpha, Sparc or the upcoming IA64. As MySQL uses a lot
of 64 bit integers internally, 64 bit CPUs will give much better
- For large databases, the optimization order is normally
RAM, Fast disks, CPU power.
- More RAM can speed up key updates by keeping most of the
used key pages in RAM.
- If you are not using transaction-safe tables or have big disks and want
to avoid long file checks, a UPS is good idea to be able to take the
system down nicely in case of a power failure.
- For systems where the database is on a dedicated server, one should
look at 1G Ethernet. Latency is as important as throughput.
- Have one dedicated disk for the system, programs and for temporary files.
If you do very many changes, put the update logs and transactions logs
on dedicated disks.
- Low seek time is important for the database disk; For big tables you can
estimate that you will need:
log(row_count) / log(index_block_length/3*2/(key_length + data_ptr_length))+1
seeks to find a row. For a table with 500,000 rows indexing a medium int:
log(500,000)/log(1024/3*2/(3+4)) +1 = 4 seeks
The above index would require: 500,000 * 7 * 3/2 = 5.2M. In real life,
most of the blocks will be buffered, so probably only 1-2 seeks are needed.
- For writes you will need (as above) 4 seek requests, however, to
find where to place the new key, and normally 2 seeks to update the
index and write the row.
- For REALLY big databases, your application will be bound by the speed of
your disk seeks, which increase by N log N as you get more data.
- Split databases and tables over different disks. In MySQL you can
use symbolic links for this.
- Striping disks (RAID 0) will increase both read and write throughput.
- Striping with mirroring (RAID 0+1) will give you safety and increase the
read speed. Write speed will be slightly lower.
- Don’t use mirroring or RAID (except RAID 0) on the disk for temporary files
or for data that can be easily re-generated..
- On Linux use
hdparm -m16 -d1 on the disks on boot to enable
reading/writing of multiple sectors at a time, and DMA. This may
increase the response time by 5-50 %.
- On Linux, mount the disks with
async (default) and
- For some specific application, one may want to have a ram disk for
some very specific tables, but normally this is not needed.
- No swap; If you have memory problems, add more RAM instead or configure
your system to use less memory.
- Don’t use NFS disks for data (you will have problems with NFS locking).
- Increase number of open files for system and for the SQL server.
ulimit -n # in the
- Increase the number of processes and threads for the system.
- If you have relatively few big tables, tell your file system to
not break up the file on different cylinders (Solaris).
- Use file systems that support big files (Solaris).
- Choose which file system to use; Reiserfs on Linux is very fast for
open, read and write. File checks take just a couple of seconds.
- Portable programs between OS and databases
- Good for quick prototyping
- One should use the DBI/DBD interface
- Simpler to learn than PERL.
- Uses less resources than PERL, which makes it good for embedding
in Web servers.
- One can get more speed by upgrading to PHP4.
- The native interface to MySQL.
- Faster and gives more control
- Lower level, so you have to work more.
- Higher level gives you more time to code your application.
- Is still in development.
- Works on Windows and Unix
- Almost portable between different SQL servers.
- Slow; MyODBC, which is a simple pass-through driver is 19 % slower
than using a native interface.
- Many ways to do the same thing; Hard to get things to work as many
ODBC drivers have different bugs in different areas.
- Problematic; Microsoft changes the interface once in a while.
- Insecure future (Microsoft pushes more for OLE than for ODBC).
- In theory portable between OS and databases.
- Can be run in the web client.
- Python + others
- May be fine, but we don’t use them.
Optimizing the application
- One should concentrate on solving the problem.
- When writing the application one should decide what is most important:
- Portability between OS
- Portability between SQL servers
- Use persistent connections.
- Cache things in your application to lessen the load of the SQL server.
- Don’t query columns that you don’t need in your application.
- Don’t use
SELECT * FROM table_name...
- Benchmark all parts of your application, but put the most effort into
benchmarking the whole application under the worst possible ‘reasonable’
load. By doing this in a modular fashion you should be able to replace the
found bottleneck with a fast ‘dummy module’, you can then easily identify
the next bottleneck (and so on).
LOCK TABLES if you do a lot of changes in a batch; For example group
Portable applications should use
- Perl DBI/DBD
- Python (or any other language that has a generalized SQL interface)
- You should only use SQL constructs which exist in all the target SQL
servers or can easily be emulated with other constructs. The crash-me
pages on http://www.mysql.com can help you with this.
- Write wrappers to provide missing functionality for other OSes / SQL servers.
If you need more speed, you should:
- Find the bottleneck (CPU, disk, memory, SQL server, OS, API, or application)
and concentrate on solving this.
- Use extensions that give you more speed / flexibility.
- Get to know your SQL server so that you can use the fastest possible
SQL constructs for your problem and avoid bottlenecks.
- Optimize your table layouts and queries.
- Use replication to get more select speed.
- If you have a slow net connection to the database, use the compressed
Don’t be afraid to make the first version of your application not
perfectly portable; when you have solved your problem, you can always
optimize it later.
- Choose compiler and compiler options.
- Find the best MySQL startup options for your system.
- Scan the the MySQL manual and read Paul DuBois’ MySQL book.
SHOW STATUS and
- Learn how the query optimizer works.
- Optimize your table formats.
- Maintain your tables (
- Use MySQL extensions to get things done faster.
- Write a MySQL UDF function if you notice that you would need some
function in many places.
- Don’t use
GRANT on table level or column level if you don’t really need it.
- Pay for MySQL support and get help to solve your problem 🙂
Compiling and installing MySQL
- By choosing the best possible compiler for your system, you can usually get
10-30 % better performance.
- On Linux/Intel, compile MySQL with pgcc. (The Pentium optimized version
of gcc). The binary will only work with Intel Pentium CPUs, however.
- Use the optimize options that are recommended in the MySQL manual for a
- Normally a native compiler for a specific CPU (like Sun Workshop for Sparc)
should give better performance than gcc, but this is not always the case.
- Compile MySQL with only the character sets you are going to use.
- Compile the mysqld executable statically
--with-mysqld-ldflags=-all-static) and strip the final executable
- Note that as MySQL doesn’t use C++ exceptions, compiling MySQL without
exceptions support will give a big performance win!
- Use native threads (instead of the included mit-pthreads) if your OS supports
- Test the resulting binary with the MySQL benchmark test.
- If possible, run
OPTIMIZE table once in a while. This is especially
important on variable size rows that are updated a lot.
- Update the key distribution statistics in your tables once in a while
myisamchk -a; Remember to take down MySQL before doing this!
- If you get fragmented files, it may be worth it to copy all files to
another disk, clear the old disk and copy the files back.
- If you have problems, check your tables with myisamchk or
- Monitor MySQL status with:
mysqladmin -i10 processlist extended-status
- With the MySQL GUI client you can monitor the process list and the status
in different windows.
mysqladmin debug to get information about locks and performance.
Use SQL for the things it’s good at, and do other things in your application.
Use the SQL server to:
- Find rows based on
Don’t use an SQL server:
- To validate data (like date)
- As a calculator
- Use keys wisely.
- Keys are good for searches, but bad for inserts / updates of key columns.
- Keep by data in the 3rd normal database form, but don’t be afraid of
duplicating information or creating summary tables if you need more
- Instead of doing a lot of
GROUP BYs on a big table, create
summary tables of the big table and query this instead.
UPDATE table set count=count+1 where key_column=constant is very fast!
- For log tables, it’s probably better to generate summary tables from them
once in a while than try to keep the summary tables live.
- Take advantage of default values on
Speed difference between different SQL servers (times in seconds)
|Reading 2000000 rows by key:
|Inserting (350768) rows:
In the above test, MySQL was run with a 8M cache; the other databases
were run with installations defaults.
Important MySQL startup options
|Change if you do a lot of new connections.
|Change if you do a lot of new connections.
|Pool for index pages; Can be made very big
|Record and key cache used by BDB tables.
|Change if you have many tables or
|Set if you need to buffer all key writes
|Find queries that takes a lot of time
ORDER BY and
|When doing a join without keys
- MySQL has a rich set of different types. You should try to use the
most efficient type for each column.
ANALYSE procedure can help you find the optimal types for a table:
SELECT * FROM table_name PROCEDURE ANALYSE()
NOT NULL for columns which will not store null values. This is
particularly important for columns which you index.
- Change your ISAM tables to MyISAM.
- If possible, create your tables with a fixed table format.
- Don’t create indexes you are not going to use.
- Use the fact that MySQL can search on a prefix of an index; If you have
INDEX (a,b), you don’t need an index on
- Instead of creating an index on long
VARCHAR column, index just a prefix
of the column to save space.
CREATE TABLE table_name (hostname CHAR(255) not null, index(hostname(10)))
- Use the most efficient table type for each table.
- Columns with identical information in different tables should be
declared identically and have identical names.
How MySQL stores data
- Databases are stored as directories.
- Tables are stored as files.
- Columns are stored in the files in dynamic length or fixed size format.
In BDB tables the data is stored in pages.
- Memory-based tables are supported.
- Databases and tables can be symbolically linked from different disks.
- On Windows MySQL supports internal symbolic links to databases with
MySQL table types
- HEAP tables; Fixed row size tables that are only stored in memory and
indexed with a HASH index.
- ISAM tables; The old B-tree table format in MySQL 3.22.
- MyISAM tables; New version of the ISAM tables with a lot of extensions:
- Binary portability.
- Index on NULL columns.
- Less fragmentation for dynamic-size rows than ISAM tables.
- Support for big files.
- Better index compression.
- Better key statistics.
- Better and faster auto_increment handling.
- Berkeley DB (BDB) tables from Sleepycat:
BEGIN WORK /
MySQL row types (only relevant for ISAM/MyISAM tables)
- MySQL will create the table in fixed size table format if all
columns are of fixed length format (no
If not, the table is created in dynamic-size format.
- Fixed-size format is much faster and more secure than the dynamic format.
- The dynamic-size row format normally takes up less space but may be
fragmented over time if the table is updated a lot.
- In some cases it’s worth it to move all
to another table just to get more speed on the main table.
pack_isam for ISAM) one can create a read-only, packed
table. This minimizes disk usage which is very nice when using slow disks.
The packed tables are perfect to use on log tables which one will not
MySQL caches (shared between all threads, allocated once)
- Key cache ; key_buffer_size, default 8M
- Table cache ; table_cache, default 64
- Thread cache ; thread_cache_size, default 0.
- Hostname cache ; Changeable at compile time, default 128.
- Memory mapped tables ; Currently only used for compressed tables.
Note that MySQL doesn’t have a row cache, but lets the OS handle this!
MySQL buffer variables (not shared, allocated on demand)
- sort_buffer ;
ORDER BY /
- record_buffer ; Scanning tables
- join_buffer_size ; Joining without keys
- myisam_sort_buffer_size ;
- net_buffer_length ; For reading the SQL statement and buffering
- tmp_table_size ; HEAP-table-size for temporary results.
How the MySQL table cache works
- Each open instance of a MyISAM table uses an index file and a data
file. If a table is used by two threads or used twice in the same query,
MyISAM will share the index file but will open another instance of the
- The cache will temporarily grow larger than the table cache size if all
tables in the cache are in use. If this happens, the next table that is
released will be closed.
- You can check if your table cache is too small by checking the mysqld
variable Opened_tables. If this value is high you should increase your
MySQL extensions / optimization that gives you speed
- Use the optimal table type (HEAP, MyISAM, or BDB tables).
- Use optimal columns for your data.
- Use fixed row size if possible.
- Use the different lock types (
REPLACE INTO table_name VALUES (...))
LOAD DATA INFILE /
- Use multi-row
INSERT to insert many rows at a time.
SELECT INTO OUTFILE
LEFT JOIN combined with
ORDER BY can use keys in some cases.
- If you only query columns that are in one index, only the index tree will
be used to resolve the query.
- Joins are normally faster than subselects (this is true for most SQL
SELECT * from table1 WHERE a > 10 LIMIT 10,20
DELETE * from table1 WHERE a > 10 LIMIT 10
foo IN (list of constants
) is very optimized.
SELECT can run concurrently.
- UDF functions that can be loaded into a running server.
- Compressed read-only tables.
CREATE TEMPORARY TABLE
CREATE TABLE .. SELECT
- MyISAM tables with RAID option to split a file over many files to get
over the 2G limit on some file system.
When MySQL uses indexes
>, >=, =, <, <=, IF NULL and BETWEEN on a key.
SELECT * FROM table_name WHERE key_part1=1 and key_part2 >
SELECT * FROM table_name WHERE key_part1 IS NULL;
- When you use a LIKE that doesn't start with a wildcard.
SELECT * FROM table_name WHERE key_part1 LIKE 'jani%'
- Retrieving rows from other tables when performing joins.
SELECT * from t1,t2 where t1.col=t2.key_part
- Find the
MIN() value for a specific index.
SELECT MIN(key_part2),MAX(key_part2) FROM table_name where key_part1=10
ORDER BY or
GROUP BY on a prefix of a key.
SELECT * FROM foo ORDER BY key_part1,key_part2,key_part3
- When all columns used in the query are part of one key.
SELECT key_part3 FROM table_name WHERE key_part1=1
When MySQL doesn’t use an index
- Indexes are NOT used if MySQL can calculate that it will probably be
faster to scan the whole table. For example if
key_part1 is evenly
distributed between 1 and 100, it’s not good to use an index in the
SELECT * FROM table_name where key_part1 > 1 and key_part1 < 90
- If you are using HEAP tables and you don’t search on all key parts with =
- When you use
ORDER BY on a HEAP table
- If you are not using the first key part
SELECT * FROM table_name WHERE key_part2=1
- If you are using
LIKE that starts with a wildcard
SELECT * FROM table_name WHERE key_part1 LIKE '%jani%'
- When you search on one index and do an ORDER BY on another
SELECT * from table_name WHERE key_part1 = # ORDER BY key2
Learn to use EXPLAIN
EXPLAIN on every query that you think is too slow!
mysql> explain select t3.DateOfAction, t1.TransactionID
-> from t1 join t2 join t3
-> where t2.ID = t1.TransactionID and t3.ID = t2.GroupID
-> order by t3.DateOfAction, t1.TransactionID;
| table | type | possible_keys | key | key_len | ref | rows | Extra |
| t1 | ALL | NULL | NULL | NULL | NULL | 11 | Using temporary; Using filesort |
| t2 | ref | ID | ID | 4 | t1.TransactionID | 13 | |
| t3 | eq_ref | PRIMARY | PRIMARY | 4 | t2.GroupID | 1 | |
range signal a potential problem.
Learn to use SHOW PROCESSLIST
SHOW processlist to find out what is going on:
| Id | User | Host | db | Command | Time | State | Info |
| 6 | monty | localhost | bp | Query | 15 | Sending data | select * from station,station as s1 |
| 8 | monty | localhost | | Query | 0 | | show processlist |
mysqladmin to kill off runaway threads.
How to find out how MySQL solves a query
Run the following commands and try to understand the output:
SHOW COLUMNS FROM ...\G
EXPLAIN SELECT ...\G
MySQL is extremely good
- For logging.
- When you do many connects; connect is very fast.
- Where you use
INSERT at the same time.
- When you don’t combine updates with selects that take a long time.
- When most selects/updates are using unique keys.
- When you use many tables without long conflicting locks.
- When you have big tables (MySQL uses a very compact table format).
Things to avoid with MySQL
- Updates to a table or
INSERT on a table with deleted rows,
SELECTS that take a long time.
HAVING on things you can have in a
JOINS without using keys or keys which are not unique enough.
JOINS on columns that have different column types.
- Using HEAP tables when not using a full key match with
- Forgetting a
WHERE clause with
DELETE in the MySQL
monitor. If you tend to do this, use the
--i-am-a-dummy option to the
Different locks in MySQL
Tricks to give MySQL more information to solve things better
Note that you can always comment out a MySQL feature to make the query portable:
SELECT /*! SQL_BUFFER_RESULTS */ ...
SELECT SQL_BUFFER_RESULTS ...
Will force MySQL to make a temporary result set. As soon as the temporary
set is done, all locks on the tables are released. This can help when
you get a problem with table locks or when it takes a long time to
transfer the result to the client.
SELECT SQL_SMALL_RESULT ... GROUP BY ...
To tell the optimizer that the result set will only contain a few rows.
SELECT SQL_BIG_RESULT ... GROUP BY ...
To tell the optimizer that the result set will contain many rows.
SELECT STRAIGHT_JOIN ...
Forces the optimizer to join the tables in the order in which they are
listed in the
SELECT ... FROM
table_name [USE INDEX (index_list) | IGNORE INDEX (index_list)] table_name2
Forces MySQL to use/ignore the listed indexes.
Example of doing transactions
- How to do a transaction with MyISAM tables:
mysql> LOCK TABLES trans READ, customer WRITE;
mysql> select sum(value) from trans where customer_id=some_id;
mysql> update customer set total_value=sum_from_previous_statement
mysql> UNLOCK TABLES;
- How to do a transaction with Berkeley DB tables:
mysql> BEGIN WORK;
mysql> select sum(value) from trans where customer_id=some_id;
mysql> update customer set total_value=sum_from_previous_statement
- Note that you can often avoid transactions altogether by doing:
UPDATE customer SET value=value+new_value WHERE customer_id=some_id;
Example of using REPLACE
REPLACE works exactly like
INSERT, except that if an old record in
the table has the same value as a new record on a unique index, the
old record is deleted before the new record is inserted.
Instead of using
SELECT 1 FROM t1 WHERE key=#
LOCK TABLES t1
DELETE FROM t1 WHERE key1=#
INSERT INTO t1 VALUES (...)
UNLOCK TABLES t1;
REPLACE INTO t1 VALUES (...)
Benefits of using MySQL 3.23
- MyISAM ; Portable BIG table format
- HEAP ; In memory tables
- Berkeley DB ; Transactional tables from Sleepycat.
- A lot of raised limits
- Dynamic character sets
GROUP BY and
LEFT JOIN ... IF NULL optimization.
CREATE TABLE ... SELECT
CREATE TEMPORARY table_name (...)
- Automatic conversion of temporary HEAP to MyISAM tables
- mysqlhotcopy script.
Important features that we are actively working on
- Improving transactions
- Fail safe replication
- Text searching
- Delete with many tables (Updates with many tables will be done after this.)
- Better key cache
RENAME TABLE foo as foo_old, foo_new as foo)
- A query cache
- A better GUI client
Author: Tim Smith
February 5, 2010 at 8:46 am (mysql)
Tags: data type, int, mysql
Over and over I see developers that don’t understand what int(11) really means. Their confusion is understandable. Many know what defining a char(10) means (a fixed-sized character string that allows up to 10 characters). However, ints are different.
First of all, there are 5 types of integer. They are all fixed size.
Type # of bytes
As you can see from the chart, an int is always 4 bytes. That can store signed numbers from -2 billion to +2 billion (and unsigned numbers 0 to 4B). So, what does it mean if you declare an int(5)? It does not restrict the number of digits to 5… It may actually do nothing! The (5) part is a display width. It’s only used if you use UNSIGNED and ZEROFILL with an integer type. Then the display of those numbers will be zero-padded on the left to 5 digits if they contain less than 5 digits. Example:
CREATE TABLE `foo` (
`bar` int(5) unsigned zerofill DEFAULT NULL
SELECT * FROM foo;
| bar |
| 00042 |
| 00101 |
| 9876543 |