Saturday, July 5, 2014

Database backup and Remove older files

find and remove files older that 7 days
find /path-to-folder/ -name "dbname_*".sql  -mtime +7 -exec rm {} \;

database backup in compressed form:

ldir=$(mysqldump --routines --single-transaction -u user -ppassword dbname | gzip > /path-to-folder/dbname_`date +"\%Y\%m\%d\%H\%M\%S"`.sql.gz);


find and remove files older that 7 days

find /path-to-folder/ -type f -a -mtime +30 -exec rm {} \;

Friday, December 27, 2013

cronnjob for backup

# vi /bin/backup.sh

#!/bin/sh
#backup.sh
ldir=$(mysqldump -u host -p pwd dbname > /test.COM/public_html/backups/dbname _`date +"\%Y\%m\%d\%H\%M\%S"`.sql);

$wq!

then scheduled cronnjob for backup for every 30 minutes 

# vi /etc/crontab

*/30 * * * * root /bin/backup.sh

$wq!

mysql locks

There are two classes of locks:

READ LOCK (SHARED LOCK)

Read locks as the name suggests, are locks taken on resources to provide consistent read by blocking writes to that resource. The read locks are also known as shared locks because many clients can acquire read locks on the same resource simultaneously. The other thing to know is that, while read lock is in place, no write lock can be acquired on that resource.

Following is how MySQL will grant read lock:

• if there is no write lock set on the resource grant the read lock immediately
• if there is a write lock on the resource, put the lock in the read lock queue

WRITE LOCK (EXCLUSIVE LOCK)

Write lock is taken on a resource when it needs to be modified. There can be only one write lock on a
resource at a given time. While a write lock is held on a resource, all other read lock requests must alsowait.

Following is how MySQL will grant write lock:

• if there is no (read/write) lock on the resource, grant the write lock immediately
• if there is a lock on the resource, put the write lock in the write lock queue

PRIORITY OF READ AND WRITE LOCK

Write lock has a higher priority than read lock. When a resource is unlocked, and if there are lock
requests waiting in the queue, then the lock is granted in the following manner:
• grant the lock first to the request waiting in the write lock queue
• if there is no lock request for the resource in the write lock queue, then grant the lock to the first
request in the read lock queue Now let’s have a brief overview of the locking mechanism of two popular table engines.

Locking in MyISAM

MyISAM supports table-level locking and does not support row-level locking. Because of this behavior, MyISAM is typically well suited for applications which have a high percentage of reads as compared to writes. But because MyISAM supports locking at a high level, the memory needed for locking is typically less, say when you compare it to row-level locks.
However, MyISAM does allow concurrent INSERTs in some situations!

CONCURRENT INSERTS

If there are no holes in MyISAM table, then INSERTs happen at the end of the table, and in such a
situation INSERTs and SELECTs are allowed to happen concurrently on the same table, with INSERT
needing no write locks. But there is a caveat, if there are many INSERT requests, then the INSERTs are done serially, which implies that only one INSERT can execute concurrently with SELECTs on the same table. Now what exactly are hole? Holes are produced when rows are deleted and/or updated in the middle of a MyISAM table.

Locking in InnoDB

InnoDB unlike MyISAM supports both table-level locking as well as row-level locking, which allows for a more fine-grained control over the resource to be locked. Row-level locking allows InnoDB to be
extremely efficient in case of heavy write-load.
InnoDB also has a very nice feature MVCC, which allows for non-locking consistent reads. This is
achieved by having different snapshots of data available to different transactions. However, if you have the transaction isolation level set to SERIALIZABLE then plain SELECTs are automatically converted to locking SELECTs. However, AUTO_INCREMENT columns need table-level locks.

TABLE-LEVEL AUTO_INC LOCKS

INSERTs into AUTO_INCREMENT column need table-level lock for the duration of the INSERT
statement. This is so that the INSERTs are safe for statement-based replication. However, as of
MySQL >= 5.1, which also introduced row-based replication, this behavior is configurable using the
variableinnodb_autoinc_lock_mode, but you still need to be using row-based replication for the INSERTs to be safe for replication. Set the value of innodb_autoinc_lock_mode to 2, and then INSERT like statements will not take any table-level locks.

InnoDB Error Codes

The following is a nonexhaustive list of common InnoDB-specific errors that you may encounter, with
information about why each occurs and how to resolve the problem.

• 1005 (ER_CANT_CREATE_TABLE)

Cannot create table. If the error message refers to error 150, table creation failed because a
foreign key constraint was not correctly formed. If the error message refers to error –1, table
creation probably failed because the table includes a column name that matched the name of an
internal InnoDB table.

• 1016 (ER_CANT_OPEN_FILE)

Cannot find the InnoDB table from the InnoDB data files, although the .frm file for the table
exists. SeeSection 14.6.13.4, “Troubleshooting InnoDB Data Dictionary Operations”.

• 1114 (ER_RECORD_FILE_FULL)

InnoDB has run out of free space in the tablespace. Reconfigure the tablespace to add a new
data file.

• 1205 (ER_LOCK_WAIT_TIMEOUT)

Lock wait timeout expired. Transaction was rolled back.

• 1206 (ER_LOCK_TABLE_FULL)

The total number of locks exceeds the lock table size. To avoid this error, increase the value
ofinnodb_buffer_pool_size. Within an individual application, a workaround may be to break
a large operation into smaller pieces. For example, if the error occurs for a large INSERT, perform
several smaller INSERToperations.

• 1213 (ER_LOCK_DEADLOCK)

Transaction deadlock. Rerun the transaction.

• 1216 (ER_NO_REFERENCED_ROW)

You are trying to add a row but there is no parent row, and a foreign key constraint fails. Add the
parent row first.

• 1217 (ER_ROW_IS_REFERENCED)

You are trying to delete a parent row that has children, and a foreign key constraint fails. Delete
the children first.

isolation levels in mysql

The four isolation levels

READ UNCOMMITTED : With the READ-UNCOMMITTED isolation level there isn’t much isolation present between the transactions at all. The transactions can see changes to data made by other transactions that are not committed yet. That means transactions could be reading data that may not even exist eventually because the other transaction that was updating the data rolled-back the changes and didn’t commit. This is known asdirty read. An application rarely needs to rely on dirty needs, in fact this really can’t be called an isolation level. Simply put, there isn’t really any isolation at all and hence such a system can’t really be called a transactional system.

READ COMMITTED : With the READ-COMMITTED isolation level, the phenomenon of dirty read is avoided, because any uncommitted changes is not visible to any other transaction, until the change is committed. Within this isolation level each SELECT uses its own snapshot of the committed data that was committed before the execution of the SELECT. Now because each SELECT has its own snapshot, so the same SELECT when run multiple times during the same transaction could return different result sets. This phenomenon is called non-repeatable read.

REPEATABLE READ : With the REPEATABLE-READ isolation level, the phenomenon of non-repeatable read is avoided. This isolation level returns the same result set through out the transaction execution for the same SELECT run any number of times. A snapshot of the SELECT is taken the first time that SELECT is run during the transaction and the same snapshot is used through out the transaction when the same SELECT is executed. A transaction running in this isolation level does not take into account any changes to data made by other transactions, regardless of whether the changes have been committed or not. This ensures that reads are always consistent(repeatable). This isolation level is the default for InnoDB. Although this isolation level solves the problem of non-repeatable read, but there is another possible problem phantom reads.

SERIALIZABLE : With the SERIALIZABLE isolation level, the phenomenon of phantom reads is avoided. Transactions when run in this isolation level place locks on all records that are accessed, as well as locks the resource so that records cannot be appended to the table being operated on by the transaction. Transactions when run in this fashion, run in a serialized manner. This isolation level is the strongest possible isolation level.

Innodb Performance Optimization

innodb_buffer_pool_size This is very important variable to tune if you’re using Innodb

tables. Innodb tables are much more sensitive to buffer size compared to MyISAM.

MyISAM may work kind of OK with default key_buffer_size even with large data set but

it will crawl with default innodb_buffer_pool_size. Also Innodb buffer pool caches both

data and index pages so you do not need to leave space for OS cache so values up to

70-80% of memory often make sense for Innodb only installations. Same rules as for

key_buffer apply – if you have small data set and it is not going to grow dramatically do

not oversize innodb_buffer_pool_size you might find better use for memory available.

innodb_additional_mem_pool_size This one does not really affect performance too

much, at least on OS with decent memory allocators. Still you might want to have it

20MB (sometimes larger) so you can see how much memory Innodb allocates for misc

needs.

innodb_log_file_size Very important for write intensive workloads especially for large

data sets. Larger sizes offer better performance but increase recovery times so be

careful. I normally use values 64M-512M depending on server size.

innodb_log_buffer_size Default for this one is kind of OK for many workloads with

medium write load and shorter transactions. If you have update activity spikes however

or work with blobs a lot you might want to increase it. Do not set it too high however

as it would be waste of memory – it is flushed every 1 sec anyway so you do not need

space for more than 1 sec worth of updates. 8MB-16MB are typically enough. Smaller

installations should use smaller values.

innodb_flush_log_at_trx_commit Crying about Innodb being 100 times slower than

MyISAM ? You probably forgot to adjust this value. Default value of 1 will mean each

update transaction commit (or each statement outside of transaction) will need to flush

log to the disk which is rather expensive, especially if you do not have Battery backed

up cache. Many applications, especially those moved from MyISAM tables are OK with

value 2 which means do not flush log to the disk but only flush it to OS cache. The log

is still flushed to the disk each second so you normally would not loose more than 1-

2 sec worth of updates. Value 0 is a bit faster but is a bit less secure as you can lose

transactions even in case MySQL Server crashes. Value 2 only cause data loss with full

OS crash.

Myisam Performance Optimization

mysql> show variables;

key_buffer_size – Very important if you use MyISAM tables. Set up to 30-40% of available memory if
you use MyISAM tables exclusively. Right size depends on amount of indexes, data size and workload
– remember MyISAM uses OS cache to cache the data so you need to leave memory for it as well, and data can be much larger than indexes in many cases.
Key_buffer - The key buffer is where MySQL caches index blocks for MyISAM tables. When a query
uses an index, MySQL checks if the index is in memory or not. Once the buffer is full, MySQL will make room for new data by replacing older data that hasn’t been used recently. Indexes are very important to database server performance that it’s hard to go wrong with making more room in memory for them.

key_buffer_size=128M

bulk_insert_buffer_size=32M

myisam_max_sort_buffer_size=8M

join_buffer_size=4M

max_allowed_packet=1M

query_cache_limit=4M

read_buffer_size=1M

read_rnd_buffer_size=2M

table_cache=128M

tmp_table_size=32M