tag:blogger.com,1999:blog-65514995762884630092024-03-13T08:54:12.246-04:00MySQL and LAMP related postsAnonymoushttp://www.blogger.com/profile/10372091392084104297noreply@blogger.comBlogger3125tag:blogger.com,1999:blog-6551499576288463009.post-33632460709990547652014-02-09T22:14:00.000-05:002014-02-10T07:40:26.999-05:00Introducing Trite: A tool for automating import of InnoDB tablespaces<div class="separator" style="clear: both; text-align: center;">
</div>
<div style="text-align: left;">
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-7c534YBvY2I/UvjIGlNCLwI/AAAAAAAAAOQ/DbytA9TlPkg/s1600/jjp2.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="http://4.bp.blogspot.com/-7c534YBvY2I/UvjIGlNCLwI/AAAAAAAAAOQ/DbytA9TlPkg/s1600/jjp2.jpg" height="320" width="292" /></a></div>
</div>
<div style="text-align: left;">
<span style="background-color: white; color: #333333; font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 15px; line-height: 25px;">Mysqldump is a fantastic tool for backing up and restoring small and medium sized MySQL tables and databases quickly. However, when </span><span style="background-color: white; color: #333333; font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 15px; line-height: 25px;">databases surge into the multi-terabyte range restoring from logical backups is inefficient. It can take a significant amount of time to insert a hundred million plus rows to a single table, even with very fast I/O. Programs like MySQL Enterprise Backup and Percona XtraBackup allow non-blocking binary copies of your InnoDB tables to be taken while it is online and processing requests. XtraBackup also has an export feature that allows InnoDB file per tablespaces to be detached from the shared table space and imported to a completely different MySQL instance.</span><br />
<span style="background-color: white; color: #333333; font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 15px; line-height: 25px;"><br />
</span> <span style="background-color: white; color: #333333; font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 15px; line-height: 25px;">The necessary steps to export and import InnoDB tables are in the XtraBackup documentation </span><a href="http://www.percona.com/doc/percona-xtrabackup/2.1/innobackupex/importing_exporting_tables_ibk.html" style="font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 15px; line-height: 25px;" target="_blank">here</a><span style="background-color: white; color: #333333; font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 15px; line-height: 25px;">. Although fairly simple and straight forward it requires a bit too much typing if you want to move more than a couple of tables. To automate the process I created a new open source project named </span><a href="https://github.com/joshuaprunier/trite" style="font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 15px; line-height: 25px;" target="_blank">Trite</a><span style="background-color: white; color: #333333; font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 15px; line-height: 25px;">.</span></div>
<br />
<span style="color: #333333; font-family: Helvetica, arial, freesans, clean, sans-serif;"><span style="font-size: 15px; line-height: 25px;">Trite stands for <b>TR</b>ansport <b>I</b>nnodb <b>T</b>ables <b>E</b>fficiently and is also a nod to the repetitive manual steps required to move tablespaces. </span></span><span style="color: #333333; font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 15px; line-height: 25px;">Trite is a client/server written in 100% <a href="http://golang.org/" target="_blank">Go</a> that leverages XtraBackups's import/export feature and aids a DBA in creating, recovering or refreshing databases. The initial problem that led me to create Trite is a common development database environment where I work. Multiple applications share the same database servers which requires each applications data be refreshed at different intervals. Before Trite we used mysqldump and it took the majority of a weekend for the data refresh to complete, even with good hardware and my.cnf's tuned for performance over crash resistance. With Trite we get the same results in a fraction of the time!</span><br />
<span style="color: #333333; font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 15px; line-height: 25px;"><br />
</span> <span style="color: #333333; font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 15px; line-height: 25px;">Trite has three modes of operation: dump, server and client. Dump mode writes out create statements for schemas, tables, stored procedures, functions, triggers and views. Server mode makes the XtraBackup and dump create files accessible via HTTP. Client mode downloads the create statements and XtraBackup files from a Trite server to recreate database objects.</span><br />
<br />
<span style="color: #333333; font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 15px; line-height: 25px;">Twelve gigabytes of </span><a href="http://imdbpy.sourceforge.net/" style="font-size: 15px; line-height: 25px;" target="_blank">IMDB</a><span style="color: #333333; font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 15px; line-height: 25px;"> data will make a suitably sized database for an example. The hardware used is a woefully under powered laptop; 4 gigs of RAM, 32bit CPU, slow SATA hard drive. Before each data load the imdb schema was removed and MySQL restarted to wipe caches. It took just over two hours to restore from the dump file. This is with innodb_doublewrite turned off to try and speed up all the inserts.</span><br />
<!-- start syntax --><br />
<div style="background: #f8f8f8; border-width: .1em .1em .1em .1em; border: solid grey; overflow: auto; padding: .2em .4em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #333333; font-family: Courier, Helvetica, arial, freesans, clean, sans-serif;"><span style="color: navy; font-weight: bold;">jprunier@beefeater:~/Downloads$</span> ls -alh imdb.sql
<span style="color: #888888;">-rw-rw-r-- 1 jprunier jprunier 4.8G Jan 24 18:41 imdb.sql</span>
<span style="color: navy; font-weight: bold;">jprunier@beefeater:~/Downloads$</span> <span style="color: green;">time </span>mysql -uroot -p < imdb.sql
<span style="color: #888888;">Enter password: </span>
<span style="color: #888888;">real 130m55.167s</span>
<span style="color: #888888;">user 2m16.052s</span>
<span style="color: #888888;">sys 0m9.020s</span>
<span style="color: navy; font-weight: bold;">jprunier@beefeater:~/Downloads$</span> sudo du -sh /var/lib/mysql/imdb
<span style="color: #888888;">[sudo] password for jprunier: </span>
<span style="color: #888888;">12G /var/lib/mysql/imdb</span>
</span></pre>
</div>
<!-- end syntax --><br />
<br />
<span style="color: #333333; font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 15px; line-height: 25px;">A binary backup of the database took 8 minutes with XtraBackup.</span><br />
<!-- start syntax --><br />
<div style="background: #f8f8f8; border-width: .1em .1em .1em .2em; border: solid grey; overflow: auto; padding: .2em .4em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #333333; font-family: Courier, Helvetica, arial, freesans, clean, sans-serif;"><span style="color: navy; font-weight: bold;">root@beefeater:~#</span> innobackupex --user<span style="color: #666666;">=</span>root --password<span style="color: #666666;">=</span> /root
<span style="color: #888888;">... output omitted ...</span>
<span style="color: #888888;">innobackupex: Backup created in directory '/root/2014-01-25_14-22-06'</span>
<span style="color: #888888;">140125 14:30:29 innobackupex: Connection to database server closed</span>
<span style="color: #888888;">140125 14:30:29 innobackupex: completed OK!</span>
<span style="color: #888888;">real 8m23.118s</span>
<span style="color: #888888;">user 0m42.076s</span>
<span style="color: #888888;">sys 0m31.524s</span>
</span></pre>
</div>
<!-- end syntax --><br />
<br />
<span style="color: #333333; font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 15px; line-height: 25px;">And 16 seconds to prepare the tables for export. This will take longer depending on the amount of changes the database receives during the backup. It can sped up by giving the process more memory.</span><br />
<!-- start syntax --><br />
<div style="background: #f8f8f8; border-width: .1em .1em .1em .2em; border: solid grey; overflow: auto; padding: .2em .4em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #333333; font-family: Courier, Helvetica, arial, freesans, clean, sans-serif;"><span style="color: navy; font-weight: bold;">root@beefeater:~#</span> <span style="color: green;">time </span>innobackupex --apply-log --use-memory<span style="color: #666666;">=</span>1G --export /root/2014-01-25_14-22-06
<span style="color: #888888;">... output omitted ...</span>
<span style="color: #888888;">140125 14:33:03 InnoDB: Starting shutdown...</span>
<span style="color: #888888;">140125 14:33:07 InnoDB: Shutdown completed; log sequence number 70808870412</span>
<span style="color: #888888;">140125 14:33:07 innobackupex: completed OK!</span>
<span style="color: #888888;">real 0m16.310s</span>
<span style="color: #888888;">user 0m0.900s</span>
<span style="color: #888888;">sys 0m0.620s</span>
</span></pre>
</div>
<!-- end syntax --><br />
<br />
<span style="color: #333333; font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 15px; line-height: 25px;">A dump of the database objects should be done as close to the backup as possible. If the Trite create statement from a dump differs significantly from the copy of table taken with XtraBackup the table may fail to import.</span><br />
<!-- start syntax --><br />
<div style="background: #f8f8f8; border-width: .1em .1em .1em .2em; border: solid grey; overflow: auto; padding: .2em .4em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #333333; font-family: Courier, Helvetica, arial, freesans, clean, sans-serif;"><span style="color: navy; font-weight: bold;">root@beefeater:~#</span> ./trite -dump -user<span style="color: #666666;">=</span>root -password<span style="color: #666666;">=</span> -host<span style="color: #666666;">=</span>localhost
<span style="color: #888888;">Dumping to: /root/localhost_dump20140125173337</span>
<span style="color: #888888;">Enter password: </span>
<span style="color: #888888;">imdb: 21 tables, 0 procedures, 0 functions, 0 triggers, 0 views</span>
<span style="color: #888888;">22 total objects dumped</span>
<span style="color: #888888;">Total runtime = 2.19331264s</span>
</span></pre>
</div>
<!-- end syntax --><br />
<br />
<span style="color: #333333; font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 15px; line-height: 25px;">In this example the Trite server is being run on the same machine as the database being restored. If your database is very large the XtraBackup, create dumps and Trite server will probably be on a different host.</span><br />
<!-- start syntax --><br />
<div style="background: #f8f8f8; border-width: .1em .1em .1em .2em; border: solid grey; overflow: auto; padding: .2em .4em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #333333; font-family: Helvetica, arial, freesans, clean, sans-serif;"><span style="color: #333333; font-family: Courier, Helvetica, arial, freesans, clean, sans-serif;"><span style="color: navy; font-weight: bold;">root@beefeater:~#</span> screen -S trite_server
<span style="color: navy; font-weight: bold;">root@beefeater:~#</span> ./trite -server -dump_path<span style="color: #666666;">=</span>/root/localhost_dump20140125173337 -backup_path<span style="color: #666666;">=</span>/root/2014-01-25_14-22-06
<span style="color: #888888;">Starting server listening on port 12000</span>
</span>
</span></pre>
</div>
<!-- end syntax --><br />
<br />
<span style="color: #333333; font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 15px; line-height: 25px;">The Trite client was able to restore the imdb database in 13 minutes. Much faster than doing all those inserts over. You can run the client with multiple worker threads, the default is single threaded. The optimal number of workers depends on the size of your tables, how fast your disks are and how fast your network is if the server is remote. In the case of the laptop, multiple workers actually makes the restore time slower.</span><br />
<!-- start syntax --><br />
<div style="background: #f8f8f8; border-width: .1em .1em .1em .2em; border: solid grey; overflow: auto; padding: .2em .4em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #333333; font-family: Courier, Helvetica, arial, freesans, clean, sans-serif;"><span style="color: navy; font-weight: bold;">root@beefeater:~#</span> ./trite -client -user<span style="color: #666666;">=</span>root -password<span style="color: #666666;">=</span> -socket<span style="color: #666666;">=</span>/var/lib/mysql/mysql.sock -server_host<span style="color: #666666;">=</span>localhost
<span style="color: #888888;">Enter password: </span>
<span style="color: #888888;">imdb.link_type has been restored</span>
<span style="color: #888888;">imdb.comp_cast_type has been restored</span>
<span style="color: #888888;">imdb.movie_info has been restored</span>
<span style="color: #888888;">imdb.aka_name has been restored</span>
<span style="color: #888888;">imdb.complete_cast has been restored</span>
<span style="color: #888888;">imdb.movie_link has been restored</span>
<span style="color: #888888;">imdb.company_name has been restored</span>
<span style="color: #888888;">imdb.char_name has been restored</span>
<span style="color: #888888;">imdb.name has been restored</span>
<span style="color: #888888;">imdb.person_info has been restored</span>
<span style="color: #888888;">imdb.keyword has been restored</span>
<span style="color: #888888;">imdb.cast_info has been restored</span>
<span style="color: #888888;">imdb.role_type has been restored</span>
<span style="color: #888888;">imdb.aka_title has been restored</span>
<span style="color: #888888;">imdb.kind_type has been restored</span>
<span style="color: #888888;">imdb.movie_info_idx has been restored</span>
<span style="color: #888888;">imdb.movie_keyword has been restored</span>
<span style="color: #888888;">imdb.company_type has been restored</span>
<span style="color: #888888;">imdb.info_type has been restored</span>
<span style="color: #888888;">imdb.title has been restored</span>
<span style="color: #888888;">imdb.movie_companies has been restored</span>
<span style="color: #888888;">Applying triggers for imdb</span>
<span style="color: #888888;">Applying views for imdb</span>
<span style="color: #888888;">Applying procedures for imdb</span>
<span style="color: #888888;">Applying functions for imdb</span>
<span style="color: #888888;">Total runtime = 13m3.881954844s</span>
</span></pre>
</div>
<!-- end syntax --><br />
<br />
<span style="color: #333333; font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 15px; line-height: 25px;">Some of the features I am planning on adding next are compression for transfer across slow networks and replication support so only objects conforming to a slaves replication filters will be applied.</span><br />
<span style="color: #333333; font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 15px; line-height: 25px;"><br />
</span> <span style="color: #333333; font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 15px; line-height: 25px;">Trite does have some limitations. The full list can be found on the </span><a href="http://github.com/joshuaprunier/trite" style="font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 15px; line-height: 25px;" target="_blank">GitHub</a><span style="color: #333333; font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 15px; line-height: 25px;"> page but the top ones are:</span><br />
<div>
<ul><span style="color: #333333; font-family: Helvetica, arial, freesans, clean, sans-serif;"><span style="color: #333333; font-family: Helvetica, arial, freesans, clean, sans-serif;"><span style="color: #333333; font-family: Helvetica, arial, freesans, clean, sans-serif;">
<li><span style="color: #333333; font-family: Helvetica, arial, freesans, clean, sans-serif;"><span style="color: #333333; font-family: Helvetica, arial, freesans, clean, sans-serif;"><span style="font-size: 15px; line-height: 25px;">InnoDB file per table is required.</span></span></span></li>
<span style="color: #333333; font-family: Helvetica, arial, freesans, clean, sans-serif;">
<li><span style="color: #333333; font-family: Helvetica, arial, freesans, clean, sans-serif;"><span style="font-size: 15px; line-height: 25px;">The database you are restoring to must be running Percona Server 5.1 or 5.5.</span></span></li>
<li><span style="color: #333333; font-family: Helvetica, arial, freesans, clean, sans-serif;"><span style="font-size: 15px; line-height: 25px;">Windows is not supported since Percona Server and XtraBackup are required.</span></span></li>
</span></span></span></span></ul>
<div>
<span style="color: #333333; font-family: Helvetica, arial, freesans, clean, sans-serif;"><span style="color: #333333; font-family: Helvetica, arial, freesans, clean, sans-serif;"><span style="color: #333333; font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 15px; line-height: 25px;">If you think Trite might benefit you and your organization give it a try and send me any feedback you might have. </span><span style="color: #333333; font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 15px; line-height: 25px;">I have tried hard to make Trite efficient and bug free but there might be issues when used with certain configurations. Be sure to test Trite on non-critical databases before use in production. </span><span style="color: #333333; font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 15px; line-height: 25px;">If you encounter any bugs please open an issue via GitHub.</span></span></span></div>
</div>
Anonymoushttp://www.blogger.com/profile/10372091392084104297noreply@blogger.com0tag:blogger.com,1999:blog-6551499576288463009.post-45627703232299515582013-06-02T21:08:00.000-04:002013-06-02T21:08:38.439-04:00DDL statements in MySQL 5.x with row-based replicationIn the replication topology I manage there are many layers of replication filters that prune data at the database and in a few places table level. The way MySQL replicates <a href="http://dev.mysql.com/doc/refman/5.5/en/sql-syntax-data-definition.html">Data Definition Language</a> (create, alter, drop) statements differs from how <a href="http://dev.mysql.com/doc/refman/5.5/en/sql-syntax-data-manipulation.html">Data Manipulation Language</a> (insert, update, delete) statements are handled with row-based replication. I often need to fix broken replication due to a lack of understanding of these subtle differences.<br />
<br />
With row-based replication DML statements focus directly on the table being modified. DDL on the other hand always uses statement-based replication and is tied to what is known in MySQL as the "default database". The default database is the schema/database currently in use when a DDL statement is executed. The only information I was able to find regarding this in the MySQL documentation is the second to last paragraph <a href="http://dev.mysql.com/doc/refman/5.5/en/binary-log-setting.html">here</a>. If do-db or ignore-db filters are configured on slaves, replication can break when DDL statements are used in certain scenarios. Table level or wildcard replication filters can also add to the potential breakage but I will save that for a separate blog post.<br />
<br />
A few examples should make things clear. The slave in the following examples is configured with "replicate-ignore-db = test". That means it will replicate all objects from its master except for anything in the "test" schema/database. The test environment is reset between examples. I have changed the prompt via the my.cnf to show the default database and statement counter: prompt = '\h(\d):\c> '<br />
<br />
<br />
<h4>
Example #1</h4>
<br />
On the master we use the test schema and create a table named ddl_test.<br />
<script src="https://gist.github.com/joshuaprunier/ea8a162b6dec0eb2d7df.js"></script><br />
<br />
The ddl_test table does not replicate to the test schema which is the expected behavior.<br />
<script src="https://gist.github.com/joshuaprunier/453005bbbe41a8013bde.js"></script><br />
<br />
<h4>
Example #2</h4>
<br />
This time on the master we use the world schema, create the ddl_test table in the test schema by way of a fully qualified table name and insert a row to it.<br />
<script src="https://gist.github.com/joshuaprunier/db72f2dd31056cb26a49.js"></script><br />
<br />
In this case the table creation is replicated because create is a DDL statement and the world schema was the default database when the statement was issued. Even though the default database was world, an insert does not replicate to the slave because it is a DML statement and the slave is configured to ignore events on objects in the test schema.<br />
<script src="https://gist.github.com/joshuaprunier/71e55e7bc88d36d65af4.js"></script><br />
<br />
<h4>
Example #3</h4>
<br />
Now we create the ddl_test table while test is the default database, then alter the table to add a column while world is the default database.<br />
<script src="https://gist.github.com/joshuaprunier/a8b3a4b41257bf94b394.js"></script><br />
<br />
Ouch! The result is broken replication on the slave. The reason is the original table create did not replicate but the additional alter statement does.<br />
<script src="https://gist.github.com/joshuaprunier/2fdee13c6da80041f12d.js"></script><br />
<br />
The error message shows exactly what went wrong. Table does not exist is pretty clear. The default database is world, the query references the test.ddl_test table and shows the alter table command that was attempted. Show slave status output also shows us the Replicate_Ignore_DB configuration has been set on the slave to ignore all DDL and DML statements against the test schema.<br />
<br />
The next course of action is to get replication going again. We have deduced the alter table statement should not have replicated so we simply skip the event using the sql_slave_skip_counter variable. <br />
<script src="https://gist.github.com/joshuaprunier/ffa1fc2f8be79d05a68e.js"></script><br />
<br />
Not directly related to the theme of this post but worth mentioning is the unique schema in MySQL known as none or null. This schema can not be used directly since it is the absence of having a default database. The only way I know of acquiring null schema is to connect to mysql without the -D option or by dropping the schema that is currently in use. How DDL and DML statements are handled remains the same with or without a default database.<br />
<br />
Hopefully the examples above make the rules that govern replication of DDL and DML statements clearer. The potential impact on a dba really depends on the size and configuration of their replication topology. The more databases and filters in place to limit and/or direct what is replicated, the greater the chance replication can break. Number of users manipulating database objects, their knowledge of MySQL and the replication topology they are working within is also a big factor.Anonymoushttp://www.blogger.com/profile/10372091392084104297noreply@blogger.com1tag:blogger.com,1999:blog-6551499576288463009.post-40011578749453376142013-05-13T18:41:00.002-04:002013-05-15T19:59:17.436-04:00Delayed row-based replication with large tables lacking a primary keyI configure all our master databases to use row-based binary logging where I work. In my opinion it is a much safer option than statement-based replication. The advantages and disadvantages of both types of MySQL replication are detailed in the online documentation <a href="https://dev.mysql.com/doc/refman/5.5/en/replication-sbr-rbr.html" rel="nofollow" target="_blank">here</a>. You can't view the events a slave is applying directly with <i><b>'show processlist'</b></i> but by issuing <i><b>'show open tables where in use'</b></i> you can detect what table is receiving the attention of the SQL thread. If you need more information the mysqlbinlog command must be used to decode the slaves relay logs or masters binary logs.<br />
<br />
Our developers often change a lot of rows with a single update statement. This usually results in some reasonable replication lag on downstream slaves. Occasionally the lag continues to grow and eventually nagios complains. Investigating the lag I sometimes discover the root of the problem is due to millions of rows updated on a table with no primary key. Putting a primary key constraint on a table is just good practice, especially on InnoDB tables. This is really necessary for any large table that will be replicated.<br />
<br />
To show what replication is actually doing I have cloned the sakila.film table, omitting the indexes, inserted all its rows and finally updated the description column for all 1,000 rows in the new table.<br />
<br />
<script src="https://gist.github.com/joshuaprunier/087e8b8383cbdb74d9fe.js"></script><br />
<br />
I have edited the output of mysqlbinlog to only show the entries related to the first row created by the insert and update statements above. The @ symbol followed by a number is a mapping to the columns in the table. So @1=film_id, @2=title, @3=description and so on. Note that the update statement records a before and after picture of the row. This is can be used in a pinch to fix mistaken updates if the damage is small instead of having to restore from backups.<br />
<br />
<script src="https://gist.github.com/joshuaprunier/f2859203ded728c9d285.js"></script><br />
<br />
So row-based replication is performing as named and creating a binary log event for each row affected. My single insert and update statement on the master then became 1,000 separate events on the slave.<br />
<br />
Digging in to the MySQL source code I was unable to confirm exactly how the SQL thread applies relay log events on a slave. I assume it is similar to what happens when a normal update statement is executed on a table with no indexes. The server must perform a full table scan to locate a single row. For a table with a million plus rows a million full table scans is expensive! A primary key or suitable unique index will prevent this type of problem.Anonymoushttp://www.blogger.com/profile/10372091392084104297noreply@blogger.com8