Batching an import [solved]

Not for 'how-to' coding questions but PHP theory instead, this forum is here for those of us who wish to learn about design aspects of programming with PHP.

Moderator: General Moderators

User avatar
VladSun
DevNet Master
Posts: 4313
Joined: Wed Jun 27, 2007 9:44 am
Location: Sofia, Bulgaria

Re: Batching an import

Post by VladSun »

Also, instead of changing mySQL config files yourself, it's better to us the preconfigured files provided by the MySQL - like my-huge.cnf, my-large.cnf, my-medium.cnf. The default MySQL installation has very limited configuration values set.
There are 10 types of people in this world, those who understand binary and those who don't
josh
DevNet Master
Posts: 4872
Joined: Wed Feb 11, 2004 3:23 pm
Location: Palm beach, Florida

Re: Batching an import

Post by josh »

I thought I was on innodb already but was mistaken, I switched it to innodb and it still failed (although took a LOT longer to fail). Trying now with my-huge.ini

Here's the log

Code: Select all

100915  1:13:47 [Note] Plugin 'FEDERATED' is disabled.
InnoDB: The first specified data file .\ibdata1 did not exist:
InnoDB: a new database to be created!
100915  1:13:47  InnoDB: Setting file .\ibdata1 size to 10 MB
InnoDB: Database physically writes the file full: wait...
100915  1:13:47  InnoDB: Log file .\ib_logfile0 did not exist: new to be created
InnoDB: Setting log file .\ib_logfile0 size to 5 MB
InnoDB: Database physically writes the file full: wait...
100915  1:13:48  InnoDB: Log file .\ib_logfile1 did not exist: new to be created
InnoDB: Setting log file .\ib_logfile1 size to 5 MB
InnoDB: Database physically writes the file full: wait...
InnoDB: Doublewrite buffer not found: creating new
InnoDB: Doublewrite buffer created
InnoDB: Creating foreign key constraint system tables
InnoDB: Foreign key constraint system tables created
100915  1:13:48  InnoDB: Started; log sequence number 0 0
100915  1:13:48 [Note] Event Scheduler: Loaded 0 events
100915  1:13:48 [Note] wampmysqld: ready for connections.
Version: '5.1.36-community-log'  socket: ''  port: 3306  MySQL Community Server (GPL)
100915  1:24:49 [Note] wampmysqld: Normal shutdown

100915  1:24:49 [Note] Event Scheduler: Purging the queue. 0 events
100915  1:24:49  InnoDB: Starting shutdown...
100915  1:24:49  InnoDB: Shutdown completed; log sequence number 0 46409
100915  1:24:49 [Warning] Forcing shutdown of 1 plugins
100915  1:24:49 [Note] wampmysqld: Shutdown complete

100915  1:27:52 [Note] Plugin 'FEDERATED' is disabled.
100915  1:27:53  InnoDB: Started; log sequence number 0 46409
100915  1:27:53 [Note] Event Scheduler: Loaded 0 events
100915  1:27:53 [Note] wampmysqld: ready for connections.
Version: '5.1.36-community-log'  socket: ''  port: 3306  MySQL Community Server (GPL)
100915  1:30:15 [Note] wampmysqld: Normal shutdown

100915  1:30:15 [Note] Event Scheduler: Purging the queue. 0 events
100915  1:30:15  InnoDB: Starting shutdown...
100915  1:30:15  InnoDB: Shutdown completed; log sequence number 0 46409
100915  1:30:15 [Warning] Forcing shutdown of 1 plugins
100915  1:30:15 [Note] wampmysqld: Shutdown complete

100915  1:30:17 [Note] Plugin 'FEDERATED' is disabled.
100915  1:30:17  InnoDB: Started; log sequence number 0 46409
100915  1:30:17 [Note] Event Scheduler: Loaded 0 events
100915  1:30:17 [Note] wampmysqld: ready for connections.
Version: '5.1.36-community-log'  socket: ''  port: 3306  MySQL Community Server (GPL)
100915  1:31:15 [Note] wampmysqld: Normal shutdown

100915  1:31:15 [Note] Event Scheduler: Purging the queue. 0 events
100915  1:31:15  InnoDB: Starting shutdown...
100915  1:31:17  InnoDB: Shutdown completed; log sequence number 0 46409
100915  1:31:17 [Warning] Forcing shutdown of 1 plugins
100915  1:31:17 [Note] wampmysqld: Shutdown complete

100915  1:31:17 [Note] Plugin 'FEDERATED' is disabled.
100915  1:31:18  InnoDB: Started; log sequence number 0 46409
100915  1:31:18 [Note] Event Scheduler: Loaded 0 events
100915  1:31:18 [Note] wampmysqld: ready for connections.
Version: '5.1.36-community-log'  socket: ''  port: 3306  MySQL Community Server (GPL)
100915  1:41:37 [Note] wampmysqld: Normal shutdown

100915  1:41:37 [Note] Event Scheduler: Purging the queue. 0 events
100915  1:41:37  InnoDB: Starting shutdown...
100915  1:41:39  InnoDB: Shutdown completed; log sequence number 0 46409
100915  1:41:39 [Warning] Forcing shutdown of 1 plugins
100915  1:41:39 [Note] wampmysqld: Shutdown complete

100915  1:41:42 [Note] Plugin 'FEDERATED' is disabled.
100915  1:41:45  InnoDB: Started; log sequence number 0 46409
100915  1:41:45 [Note] Event Scheduler: Loaded 0 events
100915  1:41:45 [Note] wampmysqld: ready for connections.
Version: '5.1.36-community-log'  socket: ''  port: 3306  MySQL Community Server (GPL)
100915  1:42:05 [Note] wampmysqld: Normal shutdown

100915  1:42:05 [Note] Event Scheduler: Purging the queue. 0 events
100915  1:42:05  InnoDB: Starting shutdown...
100915  1:42:05  InnoDB: Shutdown completed; log sequence number 0 46409
100915  1:42:05 [Warning] Forcing shutdown of 1 plugins
100915  1:42:05 [Note] wampmysqld: Shutdown complete

100915  1:42:06 [Note] Plugin 'FEDERATED' is disabled.
100915  1:42:08  InnoDB: Started; log sequence number 0 46409
100915  1:42:08 [Note] Event Scheduler: Loaded 0 events
100915  1:42:08 [Note] wampmysqld: ready for connections.
Version: '5.1.36-community-log'  socket: ''  port: 3306  MySQL Community Server (GPL)
100915  1:44:47 [Note] wampmysqld: Normal shutdown

100915  1:44:47 [Note] Event Scheduler: Purging the queue. 0 events
100915  1:44:47  InnoDB: Starting shutdown...
100915  1:44:49  InnoDB: Shutdown completed; log sequence number 0 2082343
100915  1:44:49 [Warning] Forcing shutdown of 1 plugins
100915  1:44:49 [Note] wampmysqld: Shutdown complete

100915  1:44:51 [Note] Plugin 'FEDERATED' is disabled.
100915  1:44:52  InnoDB: Started; log sequence number 0 2082343
100915  1:44:53 [Note] Event Scheduler: Loaded 0 events
100915  1:44:53 [Note] wampmysqld: ready for connections.
Version: '5.1.36-community-log'  socket: ''  port: 3306  MySQL Community Server (GPL)
100915  1:51:21 [Note] wampmysqld: Normal shutdown

100915  1:51:21 [Note] Event Scheduler: Purging the queue. 0 events
100915  1:51:23 [Warning] wampmysqld: Forcing close of thread 25  user: 'root'

100915  1:51:23  InnoDB: Starting shutdown...
100915  1:51:25  InnoDB: Shutdown completed; log sequence number 0 12005883
100915  1:51:25 [Warning] Forcing shutdown of 1 plugins
100915  1:51:25 [Note] wampmysqld: Shutdown complete

100915  1:51:26 [Note] Plugin 'FEDERATED' is disabled.
100915  1:51:27  InnoDB: Started; log sequence number 0 12005883
100915  1:51:27 [Note] Event Scheduler: Loaded 0 events
100915  1:51:27 [Note] wampmysqld: ready for connections.
Version: '5.1.36-community-log'  socket: ''  port: 3306  MySQL Community Server (GPL)
100915 16:30:42 [Note] wampmysqld: Normal shutdown

100915 16:30:42 [Note] Event Scheduler: Purging the queue. 0 events
100915 16:30:42  InnoDB: Starting shutdown...
100915 16:30:43  InnoDB: Shutdown completed; log sequence number 0 12178744
100915 16:30:43 [Warning] Forcing shutdown of 1 plugins
100915 16:30:43 [Note] wampmysqld: Shutdown complete

100915 18:23:42 [Note] Plugin 'FEDERATED' is disabled.
100915 18:23:42  InnoDB: Started; log sequence number 0 12178744
100915 18:23:42 [Note] Event Scheduler: Loaded 0 events
100915 18:23:42 [Note] wampmysqld: ready for connections.
Version: '5.1.36-community-log'  socket: ''  port: 3306  MySQL Community Server (GPL)
100915 21:49:51 [Note] wampmysqld: Normal shutdown

100915 21:49:51 [Note] Event Scheduler: Purging the queue. 0 events
100915 21:49:52  InnoDB: Starting shutdown...
100915 21:49:54  InnoDB: Shutdown completed; log sequence number 0 97052554
100915 21:49:54 [Warning] Forcing shutdown of 1 plugins
100915 21:49:54 [Note] wampmysqld: Shutdown complete

100915 21:50:00 [Note] Plugin 'FEDERATED' is disabled.
100915 21:50:01  InnoDB: Started; log sequence number 0 97052554
100915 21:50:01 [Note] Event Scheduler: Loaded 0 events
100915 21:50:01 [Note] wampmysqld: ready for connections.
Version: '5.1.36-community-log'  socket: ''  port: 3306  MySQL Community Server (GPL)
100915 21:51:39 [Note] wampmysqld: Normal shutdown

100915 21:51:39 [Note] Event Scheduler: Purging the queue. 0 events
100915 21:51:39  InnoDB: Starting shutdown...
100915 21:51:42  InnoDB: Shutdown completed; log sequence number 0 97409687
100915 21:51:42 [Warning] Forcing shutdown of 1 plugins
100915 21:51:42 [Note] wampmysqld: Shutdown complete

100915 21:51:42 [Note] Plugin 'FEDERATED' is disabled.
100915 21:51:43  InnoDB: Started; log sequence number 0 97409687
100915 21:51:43 [Note] Event Scheduler: Loaded 0 events
100915 21:51:43 [Note] wampmysqld: ready for connections.
Version: '5.1.36-community-log'  socket: ''  port: 3306  MySQL Community Server (GPL)
100917 22:50:03 [Note] wampmysqld: Normal shutdown

100917 22:50:03 [Note] Event Scheduler: Purging the queue. 0 events
100917 22:50:05 [Warning] wampmysqld: Forcing close of thread 580  user: 'root'

100918  1:50:05  InnoDB: Starting shutdown...
100918  1:50:08  InnoDB: Shutdown completed; log sequence number 0 206929986
100917 22:50:08 [Warning] Forcing shutdown of 1 plugins
100917 22:50:08 [Note] wampmysqld: Shutdown complete

100918  1:50:09 [Note] Plugin 'FEDERATED' is disabled.
100918  1:50:09  InnoDB: Started; log sequence number 0 206929986
100918  1:50:09 [Note] Event Scheduler: Loaded 0 events
100918  1:50:09 [Note] wampmysqld: ready for connections.
Version: '5.1.36-community-log'  socket: ''  port: 3306  MySQL Community Server (GPL)
100918 19:39:22 [Note] wampmysqld: Normal shutdown

100918 19:39:22 [Note] Event Scheduler: Purging the queue. 0 events
100918 19:39:24 [Warning] wampmysqld: Forcing close of thread 174  user: 'root'

100918 19:39:24  InnoDB: Starting shutdown...
100918 19:39:26  InnoDB: Shutdown completed; log sequence number 0 241271435
100918 19:39:26 [Warning] Forcing shutdown of 1 plugins
100918 19:39:26 [Note] wampmysqld: Shutdown complete

100918 19:39:28 [Note] Plugin 'FEDERATED' is disabled.
100918 19:39:28  InnoDB: Started; log sequence number 0 241271435
100918 19:39:28 [Note] Event Scheduler: Loaded 0 events
100918 19:39:28 [Note] wampmysqld: ready for connections.
Version: '5.1.36-community-log'  socket: ''  port: 3306  MySQL Community Server (GPL)
100918 19:48:35 [Note] wampmysqld: Normal shutdown

100918 19:48:35 [Note] Event Scheduler: Purging the queue. 0 events
100918 19:48:37 [Warning] wampmysqld: Forcing close of thread 10  user: 'root'

100918 19:48:37  InnoDB: Starting shutdown...
100918 19:48:39  InnoDB: Shutdown completed; log sequence number 0 244386402
100918 19:48:39 [Warning] Forcing shutdown of 1 plugins
100918 19:48:39 [Note] wampmysqld: Shutdown complete

100918 19:48:40 [Note] Plugin 'FEDERATED' is disabled.
100918 19:48:41  InnoDB: Started; log sequence number 0 244386402
100918 19:48:41 [Note] Event Scheduler: Loaded 0 events
100918 19:48:41 [Note] wampmysqld: ready for connections.
Version: '5.1.36-community-log'  socket: ''  port: 3306  MySQL Community Server (GPL)
100919 19:06:50 [Note] wampmysqld: Normal shutdown

100919 19:06:50 [Note] Event Scheduler: Purging the queue. 0 events
100919 19:06:52 [Warning] wampmysqld: Forcing close of thread 383  user: 'root'

100919 19:06:52  InnoDB: Starting shutdown...
100919 19:06:55  InnoDB: Shutdown completed; log sequence number 0 454472432
100919 19:06:55 [Warning] Forcing shutdown of 1 plugins
100919 19:06:55 [Note] wampmysqld: Shutdown complete

100919 19:07:40 [Note] Plugin 'FEDERATED' is disabled.
100919 19:07:41  InnoDB: Started; log sequence number 0 454472432
100919 19:07:41 [Note] Event Scheduler: Loaded 0 events
100919 19:07:41 [Note] wampmysqld: ready for connections.
Version: '5.1.36-community-log'  socket: ''  port: 3306  MySQL Community Server (GPL)
100919 19:07:56 [Note] wampmysqld: Normal shutdown

100919 19:07:56 [Note] Event Scheduler: Purging the queue. 0 events
100919 19:07:56  InnoDB: Starting shutdown...
100919 19:07:57  InnoDB: Shutdown completed; log sequence number 0 454472432
100919 19:07:57 [Warning] Forcing shutdown of 1 plugins
100919 19:07:57 [Note] wampmysqld: Shutdown complete

100919 19:08:12 [Note] Plugin 'FEDERATED' is disabled.
100919 19:08:12  InnoDB: Started; log sequence number 0 454472432
100919 19:08:12 [Note] Event Scheduler: Loaded 0 events
100919 19:08:12 [Note] wampmysqld: ready for connections.
Version: '5.1.36-community-log'  socket: ''  port: 3306  MySQL Community Server (GPL)
100920  0:00:19 [Note] Plugin 'FEDERATED' is disabled.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
100920  0:00:19  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Last MySQL binlog file position 0 7377860, file name .\mysql-bin.000016
100920  0:00:20  InnoDB: Started; log sequence number 0 493214085
100920  0:00:20 [Note] Recovering after a crash using mysql-bin
100920  0:00:20 [Note] Starting crash recovery...
100920  0:00:20 [Note] Crash recovery finished.
100920  0:00:21 [Note] Event Scheduler: Loaded 0 events
100920  0:00:21 [Note] wampmysqld: ready for connections.
Version: '5.1.36-community-log'  socket: ''  port: 3306  MySQL Community Server (GPL)
100921  8:31:03 [Note] wampmysqld: Normal shutdown

100921  8:31:03 [Note] Event Scheduler: Purging the queue. 0 events
100921  8:31:03  InnoDB: Starting shutdown...
100921  8:31:05  InnoDB: Shutdown completed; log sequence number 0 621057107
100921  8:31:05 [Warning] Forcing shutdown of 1 plugins
100921  8:31:05 [Note] wampmysqld: Shutdown complete

100921  8:31:07 [Note] Plugin 'FEDERATED' is disabled.
100921  8:31:08  InnoDB: Started; log sequence number 0 621057107
100921  8:31:08 [Note] Event Scheduler: Loaded 0 events
100921  8:31:08 [Note] wampmysqld: ready for connections.
Version: '5.1.36-community-log'  socket: ''  port: 3306  MySQL Community Server (GPL)
100921  8:33:32 [Note] wampmysqld: Normal shutdown

100921  8:33:32 [Note] Event Scheduler: Purging the queue. 0 events
100921  8:33:32  InnoDB: Starting shutdown...
100921  8:33:35  InnoDB: Shutdown completed; log sequence number 0 621066830
100921  8:33:35 [Warning] Forcing shutdown of 1 plugins
100921  8:33:35 [Note] wampmysqld: Shutdown complete

100921  8:33:38 [Note] Plugin 'FEDERATED' is disabled.
100921  8:33:39  InnoDB: Started; log sequence number 0 621066830
100921  8:33:40 [Note] Event Scheduler: Loaded 0 events
100921  8:33:40 [Note] wampmysqld: ready for connections.
Version: '5.1.36-community-log'  socket: ''  port: 3306  MySQL Community Server (GPL)
100921  8:34:27 [Note] wampmysqld: Normal shutdown

100921  8:34:27 [Note] Event Scheduler: Purging the queue. 0 events
100921  8:34:27  InnoDB: Starting shutdown...
100921  8:34:30  InnoDB: Shutdown completed; log sequence number 0 621069501
100921  8:34:30 [Warning] Forcing shutdown of 1 plugins
100921  8:34:30 [Note] wampmysqld: Shutdown complete

100921  8:34:31 [Note] Plugin 'FEDERATED' is disabled.
100921  8:34:33  InnoDB: Started; log sequence number 0 621069501
100921  8:34:33 [Note] Event Scheduler: Loaded 0 events
100921  8:34:33 [Note] wampmysqld: ready for connections.
Version: '5.1.36-community-log'  socket: ''  port: 3306  MySQL Community Server (GPL)
100921  8:36:36 [Note] wampmysqld: Normal shutdown

100921  8:36:36 [Note] Event Scheduler: Purging the queue. 0 events
100921  8:36:36  InnoDB: Starting shutdown...
100921  8:36:39  InnoDB: Shutdown completed; log sequence number 0 621075589
100921  8:36:39 [Warning] Forcing shutdown of 1 plugins
100921  8:36:39 [Note] wampmysqld: Shutdown complete

100921  8:36:43 [Note] Plugin 'FEDERATED' is disabled.
100921  8:36:46  InnoDB: Started; log sequence number 0 621075589
100921  8:36:47 [Note] Event Scheduler: Loaded 0 events
100921  8:36:47 [Note] wampmysqld: ready for connections.
Version: '5.1.36-community-log'  socket: ''  port: 3306  MySQL Community Server (GPL)
100921  8:37:50 [Note] wampmysqld: Normal shutdown

100921  8:37:50 [Note] Event Scheduler: Purging the queue. 0 events
100921  8:37:50  InnoDB: Starting shutdown...
100921  8:37:51  InnoDB: Shutdown completed; log sequence number 0 621075628
100921  8:37:51 [Warning] Forcing shutdown of 1 plugins
100921  8:37:51 [Note] wampmysqld: Shutdown complete

100921  8:37:52 [Note] Plugin 'FEDERATED' is disabled.
100921  8:37:53  InnoDB: Started; log sequence number 0 621075628
100921  8:37:53 [Note] Event Scheduler: Loaded 0 events
100921  8:37:53 [Note] wampmysqld: ready for connections.
Version: '5.1.36-community-log'  socket: ''  port: 3306  MySQL Community Server (GPL)
100922  5:19:12 [Note] wampmysqld: Normal shutdown

100922  5:19:12 [Note] Event Scheduler: Purging the queue. 0 events
100922  5:19:14  InnoDB: Starting shutdown...
100922  5:19:17  InnoDB: Shutdown completed; log sequence number 0 754026468
100922  5:19:17 [Warning] Forcing shutdown of 1 plugins
100922  5:19:17 [Note] wampmysqld: Shutdown complete

100922  5:19:43 [Note] Plugin 'FEDERATED' is disabled.
100922  5:19:43  InnoDB: Started; log sequence number 0 754026468
100922  5:19:43 [Note] Event Scheduler: Loaded 0 events
100922  5:19:43 [Note] wampmysqld: ready for connections.
Version: '5.1.36-community-log'  socket: ''  port: 3306  MySQL Community Server (GPL)
100924  3:38:42 [Note] wampmysqld: Normal shutdown

100924  3:38:42 [Note] Event Scheduler: Purging the queue. 0 events
100924  3:38:44 [Warning] wampmysqld: Forcing close of thread 1255  user: 'root'

100924  3:38:44  InnoDB: Starting shutdown...
100924  3:38:46  InnoDB: Shutdown completed; log sequence number 0 1200255512
100924  3:38:46 [Warning] Forcing shutdown of 1 plugins
100924  3:38:46 [Note] wampmysqld: Shutdown complete

100924  3:38:47 [Note] Plugin 'FEDERATED' is disabled.
100924  3:38:48  InnoDB: Started; log sequence number 0 1200255512
100924  3:38:48 [Note] Event Scheduler: Loaded 0 events
100924  3:38:48 [Note] wampmysqld: ready for connections.
Version: '5.1.36-community-log'  socket: ''  port: 3306  MySQL Community Server (GPL)
100924  3:53:34 [Note] wampmysqld: Normal shutdown

100924  3:53:34 [Note] Event Scheduler: Purging the queue. 0 events
100924  3:53:36 [Warning] wampmysqld: Forcing close of thread 3  user: 'root'

100924  3:53:36 [Warning] wampmysqld: Forcing close of thread 2  user: 'root'

100924  3:53:36  InnoDB: Starting shutdown...
100924  3:53:39  InnoDB: Shutdown completed; log sequence number 0 1200447298
100924  3:53:39 [Warning] Forcing shutdown of 1 plugins
100924  3:53:39 [Note] wampmysqld: Shutdown complete

100924  3:53:39 [Note] Plugin 'FEDERATED' is disabled.
100924  3:53:40  InnoDB: Started; log sequence number 0 1200447298
100924  3:53:40 [Note] Event Scheduler: Loaded 0 events
100924  3:53:40 [Note] wampmysqld: ready for connections.
Version: '5.1.36-community-log'  socket: ''  port: 3306  MySQL Community Server (GPL)
100924  4:06:00 [Note] wampmysqld: Normal shutdown

100924  4:06:00 [Note] Event Scheduler: Purging the queue. 0 events
100924  4:06:02 [Warning] wampmysqld: Forcing close of thread 2  user: 'root'

100924  4:06:02 [Warning] wampmysqld: Forcing close of thread 1  user: 'root'

100924  4:06:02  InnoDB: Starting shutdown...
100924  4:06:05  InnoDB: Shutdown completed; log sequence number 0 1200639522
100924  4:06:05 [Warning] Forcing shutdown of 1 plugins
100924  4:06:05 [Note] wampmysqld: Shutdown complete

100924  4:06:06 [Note] Plugin 'FEDERATED' is disabled.
100924  4:06:06  InnoDB: Started; log sequence number 0 1200639522
100924  4:06:06 [Note] Event Scheduler: Loaded 0 events
100924  4:06:06 [Note] wampmysqld: ready for connections.
Version: '5.1.36-community-log'  socket: ''  port: 3306  MySQL Community Server (GPL)
100924  4:06:15 [Note] wampmysqld: Normal shutdown

100924  4:06:15 [Note] Event Scheduler: Purging the queue. 0 events
100924  4:06:15  InnoDB: Starting shutdown...
100924  4:06:15  InnoDB: Shutdown completed; log sequence number 0 1200639522
100924  4:06:15 [Warning] Forcing shutdown of 1 plugins
100924  4:06:15 [Note] wampmysqld: Shutdown complete

100924  4:06:16 [Note] Plugin 'FEDERATED' is disabled.
100924  4:06:16  InnoDB: Started; log sequence number 0 1200639522
100924  4:06:16 [Note] Event Scheduler: Loaded 0 events
100924  4:06:16 [Note] wampmysqld: ready for connections.
Version: '5.1.36-community-log'  socket: ''  port: 3306  MySQL Community Server (GPL)
100924  4:07:53 [Note] wampmysqld: Normal shutdown

100924  4:07:53 [Note] Event Scheduler: Purging the queue. 0 events
100924  4:07:53  InnoDB: Starting shutdown...
100924  4:07:54  InnoDB: Shutdown completed; log sequence number 0 1200639522
100924  4:07:54 [Warning] Forcing shutdown of 1 plugins
100924  4:07:54 [Note] wampmysqld: Shutdown complete

100924  4:07:54 [Note] Plugin 'FEDERATED' is disabled.
100924  4:07:54  InnoDB: Started; log sequence number 0 1200639522
100924  4:07:54 [Note] Event Scheduler: Loaded 0 events
100924  4:07:54 [Note] wampmysqld: ready for connections.
Version: '5.1.36-community-log'  socket: ''  port: 3306  MySQL Community Server (GPL)
100924  4:08:11 [Note] wampmysqld: Normal shutdown

100924  4:08:11 [Note] Event Scheduler: Purging the queue. 0 events
100924  4:08:11  InnoDB: Starting shutdown...
100924  4:08:12  InnoDB: Shutdown completed; log sequence number 0 1200639522
100924  4:08:12 [Warning] Forcing shutdown of 1 plugins
100924  4:08:12 [Note] wampmysqld: Shutdown complete

100924  4:08:12 [Note] Plugin 'FEDERATED' is disabled.
100924  4:08:13  InnoDB: Started; log sequence number 0 1200639522
100924  4:08:13 [Note] Event Scheduler: Loaded 0 events
100924  4:08:13 [Note] wampmysqld: ready for connections.
Version: '5.1.36-community-log'  socket: ''  port: 3306  MySQL Community Server (GPL)
100924  4:25:46 [Note] wampmysqld: Normal shutdown

100924  4:25:46 [Note] Event Scheduler: Purging the queue. 0 events
100924  4:25:48 [Warning] wampmysqld: Forcing close of thread 18  user: 'root'

100924  4:25:48  InnoDB: Starting shutdown...
100924  4:25:49  InnoDB: Shutdown completed; log sequence number 0 1203726724
100924  4:25:49 [Warning] Forcing shutdown of 1 plugins
100924  4:25:49 [Note] wampmysqld: Shutdown complete

100924  4:26:24 [Note] Plugin 'FEDERATED' is disabled.
100924  4:26:24  InnoDB: Started; log sequence number 0 1203726724
100924  4:26:24 [Note] Event Scheduler: Loaded 0 events
100924  4:26:24 [Note] wampmysqld: ready for connections.
Version: '5.1.36-community-log'  socket: ''  port: 3306  MySQL Community Server (GPL)
100924  5:57:28 [Note] wampmysqld: Normal shutdown

100924  5:57:28 [Note] Event Scheduler: Purging the queue. 0 events
100924  5:57:30 [Warning] wampmysqld: Forcing close of thread 88  user: 'root'

100924  5:57:30  InnoDB: Starting shutdown...
100924  5:57:31  InnoDB: Shutdown completed; log sequence number 0 1239378019
100924  5:57:31 [Warning] Forcing shutdown of 1 plugins
100924  5:57:31 [Note] wampmysqld: Shutdown complete

100924  5:57:48 [Note] Plugin 'FEDERATED' is disabled.
100924  5:57:49  InnoDB: Started; log sequence number 0 1239378019
100924  5:57:49 [Note] Event Scheduler: Loaded 0 events
100924  5:57:49 [Note] wampmysqld: ready for connections.
Version: '5.1.36-community-log'  socket: ''  port: 3306  MySQL Community Server (GPL)
100924  6:00:43 [Note] wampmysqld: Normal shutdown

100924  6:00:43 [Note] Event Scheduler: Purging the queue. 0 events
100924  6:00:43  InnoDB: Starting shutdown...
100924  6:00:46  InnoDB: Shutdown completed; log sequence number 0 1239598738
100924  6:00:46 [Warning] Forcing shutdown of 1 plugins
100924  6:00:46 [Note] wampmysqld: Shutdown complete

100924  6:00:47 [Note] Plugin 'FEDERATED' is disabled.
100924  6:00:48  InnoDB: Started; log sequence number 0 1239598738
100924  6:00:48 [Note] Event Scheduler: Loaded 0 events
100924  6:00:48 [Note] wampmysqld: ready for connections.
Version: '5.1.36-community-log'  socket: ''  port: 3306  MySQL Community Server (GPL)
100924  6:14:56 [Note] wampmysqld: Normal shutdown

100924  6:14:56 [Note] Event Scheduler: Purging the queue. 0 events
100924  6:14:58 [Warning] wampmysqld: Forcing close of thread 7  user: 'root'

100924  6:14:58  InnoDB: Starting shutdown...
100924  6:15:01  InnoDB: Shutdown completed; log sequence number 0 1239946431
100924  6:15:01 [Warning] Forcing shutdown of 1 plugins
100924  6:15:01 [Note] wampmysqld: Shutdown complete

100924  6:15:01 [Note] Plugin 'FEDERATED' is disabled.
100924  6:15:02  InnoDB: Started; log sequence number 0 1239946431
100924  6:15:02 [Note] Event Scheduler: Loaded 0 events
100924  6:15:02 [Note] wampmysqld: ready for connections.
Version: '5.1.36-community-log'  socket: ''  port: 3306  MySQL Community Server (GPL)
100924  6:24:56 [Note] wampmysqld: Normal shutdown

100924  6:24:56 [Note] Event Scheduler: Purging the queue. 0 events
100924  6:24:58 [Warning] wampmysqld: Forcing close of thread 3  user: 'root'

100924  6:24:58  InnoDB: Starting shutdown...
100924  6:25:01  InnoDB: Shutdown completed; log sequence number 0 1240141365
100924  6:25:01 [Warning] Forcing shutdown of 1 plugins
100924  6:25:01 [Note] wampmysqld: Shutdown complete

100924  6:25:01 [Note] Plugin 'FEDERATED' is disabled.
100924  6:25:02  InnoDB: Started; log sequence number 0 1240141365
100924  6:25:02 [Note] Event Scheduler: Loaded 0 events
100924  6:25:02 [Note] wampmysqld: ready for connections.
Version: '5.1.36-community-log'  socket: ''  port: 3306  MySQL Community Server (GPL)
100924  8:23:47 [Note] wampmysqld: Normal shutdown

100924  8:23:47 [Note] Event Scheduler: Purging the queue. 0 events
100924  8:23:49 [Warning] wampmysqld: Forcing close of thread 5  user: 'root'

100924  8:23:50  InnoDB: Starting shutdown...
100924  8:23:52  InnoDB: Shutdown completed; log sequence number 0 1295307649
100924  8:23:52 [Warning] Forcing shutdown of 1 plugins
100924  8:23:52 [Note] wampmysqld: Shutdown complete

I should also point out, the query continues running - even though the PHP script dies.
mikosiko
Forum Regular
Posts: 757
Joined: Wed Jan 13, 2010 7:22 pm

Re: Batching an import

Post by mikosiko »

in your php.ini what value has the max_execution_time directive?
josh
DevNet Master
Posts: 4872
Joined: Wed Feb 11, 2004 3:23 pm
Location: Palm beach, Florida

Re: Batching an import

Post by josh »

Same issue with my-huge.cnf

Code: Select all

Fatal error: Uncaught exception 'Zend_Db_Statement_Exception' with message 'SQLSTATE[HY000]: General error: 2006 MySQL server has gone away' in E:\dev\vaf\lib\Z
end\Db\Statement\Pdo.php:234
Stack trace:
#0 E:\dev\vaf\lib\Zend\Db\Statement.php(300): Zend_Db_Statement_Pdo->_execute(Array)
#1 E:\dev\vaf\lib\Zend\Db\Adapter\Abstract.php(468): Zend_Db_Statement->execute(Array)
#2 E:\dev\vaf\lib\Zend\Db\Adapter\Pdo\Abstract.php(238): Zend_Db_Adapter_Abstract->query('UPDATE elite_im...', Array)
sike
Forum Commoner
Posts: 84
Joined: Wed Aug 02, 2006 8:33 am

Re: Batching an import

Post by sike »

You need to be careful creating indices on each column because that might slowdown insertion of records big time. Best thing to start is examining the queries you issue with EXPLAIN. That will give you a idea what mysql tries todo and why it's slow. If you have never used EXPLAIN if found a quite good presentation : http://forge.mysql.com/w/images/3/3f/My ... xplain.pdf which should get you started.

cheers
Chris
User avatar
VladSun
DevNet Master
Posts: 4313
Joined: Wed Jun 27, 2007 9:44 am
Location: Sofia, Bulgaria

Re: Batching an import

Post by VladSun »

josh wrote:Ok - same problem - dead after 60 seconds exactly
The only config item with value of 60 is the net_write_timeout
Try changing it ... Though, "MySQL server has gone away" error is usually caused by a query longer than the max_allowed_packet.
There are 10 types of people in this world, those who understand binary and those who don't
josh
DevNet Master
Posts: 4872
Joined: Wed Feb 11, 2004 3:23 pm
Location: Palm beach, Florida

Re: Batching an import

Post by josh »

Did that, same issue.

Code: Select all

+----------------------------+-------+
| Variable_name              | Value |
+----------------------------+-------+
| connect_timeout            | 99999 |
| delayed_insert_timeout     | 300   |
| innodb_lock_wait_timeout   | 999   |
| innodb_rollback_on_timeout | OFF   |
| interactive_timeout        | 28800 |
| net_read_timeout           | 99999 |
| net_write_timeout          | 99999 |
| slave_net_timeout          | 3600  |
| table_lock_wait_timeout    | 999   |
| wait_timeout               | 28800 |
+----------------------------+-------+
10 rows in set (0.00 sec)
To the person suggesting max_execution_time - do you see the error message I am getting?

To the person suggesting explain - the query continues to run - it doesnt matter how effective or unaffective my query is. The mysql connection is being dropped on the *PHP* side after exactly 60 seconds. The queries in the server are running great. Structuring the queries such that they don't exceed 60 seconds is NOT an acceptable solution, sorry.. The error would still be reproduced then on larger data sets or smaller servers.

I'm going to play around with some PDO client settings next.
josh
DevNet Master
Posts: 4872
Joined: Wed Feb 11, 2004 3:23 pm
Location: Palm beach, Florida

Re: Batching an import

Post by josh »

Important Update: Happens on php5.3 (crash) but does not on php5.2 (stays connected well over 170 seconds)

php 5.3 phpinfo

Code: Select all


E:\dev\vaf\app\code\local\Elite\Vafimporter\cron>C:\wamp\bin\php\php5.3.0\php -i 
phpinfo()
PHP Version => 5.3.0

System => Windows NT COMPUTER_1 5.1 build 2600 (Windows XP Professional Service Pack 3) i586
Build Date => Jun 29 2009 21:23:30
Compiler => MSVC6 (Visual C++ 6.0)
Architecture => x86
Configure Command => cscript /nologo configure.js  "--enable-snapshot-build" "--disable-isapi" "--enable-debug-pack" "--with-pdo-oci=D:\php-sdk\oracle\instantclient10\sdk,shared" "--with-oci8=D:\php-sdk\oracle\instantclient10\sdk,shared" "--with-oci8-11g=D:\php-sdk\oracle\instantclient11\sdk,shared" "--with-enchant=shared"
Server API => Command Line Interface
Virtual Directory Support => enabled
Configuration File (php.ini) Path => C:\WINDOWS
Loaded Configuration File => C:\wamp\bin\php\php5.3.0\php.ini
Scan this dir for additional .ini files => (none)
Additional .ini files parsed => (none)
PHP API => 20090626
PHP Extension => 20090626
Zend Extension => 220090626
Zend Extension Build => API220090626,TS,VC6
PHP Extension Build => API20090626,TS,VC6
Debug Build => no
Thread Safety => enabled
Zend Memory Manager => enabled
Zend Multibyte Support => disabled
IPv6 Support => enabled
Registered PHP Streams => php, file, glob, data, http, ftp, zip, compress.zlib, phar  
Registered Stream Socket Transports => tcp, udp
Registered Stream Filters => convert.iconv.*, string.rot13, string.toupper, string.tolower, string.strip_tags, convert.*, consumed, dechunk, zlib.*


This program makes use of the Zend Scripting Language Engine:
Zend Engine v2.3.0, Copyright (c) 1998-2009 Zend Technologies
    with DBG v3.9.8, (C) 2000,2010, by Dmitri Dmitrienko


 _______________________________________________________________________


Configuration

bcmath

BCMath support => enabled

Directive => Local Value => Master Value
bcmath.scale => 0 => 0

calendar

Calendar support => enabled

com_dotnet

COM support => enabled
DCOM support => disabled
.Net support => not present in this build

Directive => Local Value => Master Value
com.allow_dcom => 0 => 0
com.autoregister_casesensitive => 1 => 1
com.autoregister_typelib => 0 => 0
com.autoregister_verbose => 0 => 0
com.code_page => no value => no value
com.typelib_file => no value => no value

Core

PHP Version => 5.3.0

Directive => Local Value => Master Value
allow_call_time_pass_reference => Off => Off
allow_url_fopen => On => On
allow_url_include => Off => Off
always_populate_raw_post_data => Off => Off
arg_separator.input => & => &
arg_separator.output => & => &
asp_tags => Off => Off
auto_append_file => no value => no value
auto_globals_jit => On => On
auto_prepend_file => no value => no value
browscap => no value => no value
default_charset => no value => no value
default_mimetype => text/html => text/html
define_syslog_variables => Off => Off
disable_classes => no value => no value
disable_functions => no value => no value
display_errors => STDOUT => STDOUT
display_startup_errors => On => On
doc_root => no value => no value
docref_ext => no value => no value
docref_root => no value => no value
enable_dl => Off => Off
error_append_string => no value => no value
error_log => no value => no value
error_prepend_string => no value => no value
error_reporting => 30719 => 30719
exit_on_timeout => Off => Off
expose_php => On => On
extension_dir => c:/wamp/bin/php/php5.3.0/ext/ => c:/wamp/bin/php/php5.3.0/ext/
file_uploads => On => On
highlight.bg => <font style="color: #FFFFFF">#FFFFFF</font> => <font style="color: #FFFFFF">#FFFFFF</font>
highlight.comment => <font style="color: #FF8000">#FF8000</font> => <font style="color: #FF8000">#FF8000</font>
highlight.default => <font style="color: #0000BB">#0000BB</font> => <font style="color: #0000BB">#0000BB</font>
highlight.html => <font style="color: #000000">#000000</font> => <font style="color: #000000">#000000</font>
highlight.keyword => <font style="color: #007700">#007700</font> => <font style="color: #007700">#007700</font>
highlight.string => <font style="color: #DD0000">#DD0000</font> => <font style="color: #DD0000">#DD0000</font>
html_errors => Off => Off
ignore_repeated_errors => Off => Off
ignore_repeated_source => Off => Off
ignore_user_abort => Off => Off
implicit_flush => On => On
include_path => .;C:\wamp\bin\php\php5.3.0\pear => .;C:\wamp\bin\php\php5.3.0\pear
log_errors => On => On
log_errors_max_len => 1024 => 1024
magic_quotes_gpc => Off => Off
magic_quotes_runtime => Off => Off
magic_quotes_sybase => Off => Off
mail.add_x_header => On => On
mail.force_extra_parameters => no value => no value
mail.log => no value => no value
max_execution_time => 0 => 0
max_input_nesting_level => 64 => 64
max_input_time => -1 => -1
memory_limit => 128M => 128M
open_basedir => no value => no value
output_buffering => 0 => 0
output_handler => no value => no value
post_max_size => 8M => 8M
precision => 14 => 14
realpath_cache_size => 16K => 16K
realpath_cache_ttl => 120 => 120
register_argc_argv => On => On
register_globals => Off => Off
register_long_arrays => Off => Off
report_memleaks => On => On
report_zend_debug => Off => Off
request_order => GP => GP
safe_mode => Off => Off
safe_mode_exec_dir => no value => no value
safe_mode_gid => Off => Off
safe_mode_include_dir => no value => no value
sendmail_from => you@yourdomain => you@yourdomain
sendmail_path => no value => no value
serialize_precision => 100 => 100
short_open_tag => Off => Off
SMTP => localhost => localhost
smtp_port => 25 => 25
sql.safe_mode => Off => Off
track_errors => On => On
unserialize_callback_func => no value => no value
upload_max_filesize => 2M => 2M
upload_tmp_dir => c:/wamp/tmp => c:/wamp/tmp
user_dir => no value => no value
user_ini.cache_ttl => 300 => 300
user_ini.filename => .user.ini => .user.ini
variables_order => GPCS => GPCS
xmlrpc_error_number => 0 => 0
xmlrpc_errors => Off => Off
y2k_compliance => On => On
zend.enable_gc => On => On

ctype

ctype functions => enabled

date

date/time support => enabled
"Olson" Timezone Database Version => 2009.10
Timezone Database => internal
Default timezone => UTC

Directive => Local Value => Master Value
date.default_latitude => 31.7667 => 31.7667
date.default_longitude => 35.2333 => 35.2333
date.sunrise_zenith => 90.583333 => 90.583333
date.sunset_zenith => 90.583333 => 90.583333
date.timezone => UTC => UTC

dbg

Version => 3.9.8
Linked => as a shared library.
Dbg SSL => not available
Dbg ip security => enabled.
Profiler => enabled
Target PHP version => 5.3.x
Server API => Command Line Interface
Extensions API => 220090626
Modules API => 20090626
PHP API => 20090626

Directive => Local Value => Master Value
debugger.enable_session_cookie => On => On
debugger.enabled => On => On
debugger.fail_silently => Off => Off
debugger.hosts_allow => <i>protected value</i> => <i>protected value</i>
debugger.hosts_deny => <i>protected value</i> => <i>protected value</i>
debugger.ignore_nops => Off => Off
debugger.JIT_enabled => Off => Off
debugger.JIT_host => clienthost => clienthost
debugger.JIT_level => 3 => 3
debugger.JIT_port => 7869 => 7869
debugger.net.chunksize => 48000 => 48000
debugger.net.maxpacketsize => 50000000 => 50000000
debugger.ports => <i>protected value</i> => <i>protected value</i>
debugger.profiler_enabled => On => On
debugger.session_nocache => On => On
debugger.timeout_seconds => 300 => 300

dom

DOM/XML => enabled
DOM/XML API Version => 20031129
libxml Version => 2.7.3
HTML Support => enabled
XPath Support => enabled
XPointer Support => enabled
Schema Support => enabled
RelaxNG Support => enabled

ereg

Regex Library => Bundled library enabled

filter

Input Validation and Filtering => enabled
Revision => $Revision: 1.52.2.39.2.16 $

Directive => Local Value => Master Value
filter.default => unsafe_raw => unsafe_raw
filter.default_flags => no value => no value

ftp

FTP support => enabled

gd

GD Support => enabled
GD Version => bundled (2.0.34 compatible)
FreeType Support => enabled
FreeType Linkage => with freetype
FreeType Version => 2.3.9
GIF Read Support => enabled
GIF Create Support => enabled
JPEG Support => enabled
libJPEG Version => 6b
PNG Support => enabled
libPNG Version => 1.2.37
WBMP Support => enabled
XBM Support => enabled

Directive => Local Value => Master Value
gd.jpeg_ignore_warning => 0 => 0

hash

hash support => enabled
Hashing Engines => md2 md4 md5 sha1 sha224 sha256 sha384 sha512 ripemd128 ripemd160 ripemd256 ripemd320 whirlpool tiger128,3 tiger160,3 tiger192,3 tiger128,4 tiger160,4 tiger192,4 snefru snefru256 gost adler32 crc32 crc32b salsa10 salsa20 haval128,3 haval160,3 haval192,3 haval224,3 haval256,3 haval128,4 haval160,4 haval192,4 haval224,4 haval256,4 haval128,5 haval160,5 haval192,5 haval224,5 haval256,5 

iconv

iconv support => enabled
iconv implementation => "libiconv"
iconv library version => 1.11

Directive => Local Value => Master Value
iconv.input_encoding => ISO-8859-1 => ISO-8859-1
iconv.internal_encoding => ISO-8859-1 => ISO-8859-1
iconv.output_encoding => ISO-8859-1 => ISO-8859-1

json

json support => enabled
json version => 1.2.1

libxml

libXML support => active
libXML Compiled Version => 2.7.3
libXML Loaded Version => 20703
libXML streams => enabled

mbstring

Multibyte Support => enabled
Multibyte string engine => libmbfl
HTTP input encoding translation => disabled

mbstring extension makes use of "streamable kanji code filter and converter", which is distributed under the GNU Lesser General Public License version 2.1.

Multibyte (japanese) regex support => enabled
Multibyte regex (oniguruma) version => 4.7.1

Directive => Local Value => Master Value
mbstring.detect_order => no value => no value
mbstring.encoding_translation => Off => Off
mbstring.func_overload => 0 => 0
mbstring.http_input => pass => pass
mbstring.http_output => pass => pass
mbstring.http_output_conv_mimetypes => ^(text/|application/xhtml\+xml) => ^(text/|application/xhtml\+xml)
mbstring.internal_encoding => no value => no value
mbstring.language => neutral => neutral
mbstring.strict_detection => Off => Off
mbstring.substitute_character => no value => no value

mcrypt

mcrypt support => enabled
Version => 2.5.8
Api No => 20021217
Supported ciphers => cast-128 gost rijndael-128 twofish cast-256 loki97 rijndael-192 saferplus wake blowfish-compat des rijndael-256 serpent xtea blowfish enigma rc2 tripledes arcfour 
Supported modes => cbc cfb ctr ecb ncfb nofb ofb stream 

Directive => Local Value => Master Value
mcrypt.algorithms_dir => no value => no value
mcrypt.modes_dir => no value => no value

mhash

MHASH support => Enabled
MHASH API Version => Emulated Support

mysql

MySQL Support => enabled
Active Persistent Links => 0
Active Links => 0
Client API version => mysqlnd 5.0.5-dev - 081106 - $Revision: 1.3.2.27 $
Persistent cache => enabled
put_hits => 0
put_misses => 0
get_hits => 0
get_misses => 0
size => 2000
free_items => 2000
references => 2

Directive => Local Value => Master Value
mysql.allow_local_infile => On => On
mysql.allow_persistent => On => On
mysql.cache_size => 2000 => 2000
mysql.connect_timeout => 60 => 60
mysql.default_host => no value => no value
mysql.default_password => no value => no value
mysql.default_port => no value => no value
mysql.default_socket => no value => no value
mysql.default_user => no value => no value
mysql.max_links => Unlimited => Unlimited
mysql.max_persistent => Unlimited => Unlimited
mysql.trace_mode => Off => Off

mysqli

MysqlI Support => enabled
Client API library version => mysqlnd 5.0.5-dev - 081106 - $Revision: 1.3.2.27 $
Active Persistent Links => 0
Inactive Persistent Links => 0
Active Links => 0
Persistent cache => enabled
put_hits => 0
put_misses => 0
get_hits => 0
get_misses => 0
size => 2000
free_items => 2000
references => 2

Directive => Local Value => Master Value
mysqli.allow_local_infile => On => On
mysqli.allow_persistent => On => On
mysqli.cache_size => 2000 => 2000
mysqli.default_host => no value => no value
mysqli.default_port => 3306 => 3306
mysqli.default_pw => no value => no value
mysqli.default_socket => no value => no value
mysqli.default_user => no value => no value
mysqli.max_links => Unlimited => Unlimited
mysqli.max_persistent => Unlimited => Unlimited
mysqli.reconnect => Off => Off

mysqlnd

mysqlnd => enabled
Version => mysqlnd 5.0.5-dev - 081106 - $Revision: 1.3.2.27 $
Command buffer size => 2048
Read buffer size => 32768
Collecting statistics => Yes
Collecting memory statistics => Yes

Client statistics =>  
bytes_sent => 0
bytes_received => 0
packets_sent => 0
packets_received => 0
protocol_overhead_in => 0
protocol_overhead_out => 0
bytes_received_ok_packet => 0
bytes_received_eof_packet => 0
bytes_received_rset_header_packet => 0
bytes_received_rset_field_meta_packet => 0
bytes_received_rset_row_packet => 0
bytes_received_prepare_response_packet => 0
bytes_received_change_user_packet => 0
packets_sent_command => 0
packets_received_ok => 0
packets_received_eof => 0
packets_received_rset_header => 0
packets_received_rset_field_meta => 0
packets_received_rset_row => 0
packets_received_prepare_response => 0
packets_received_change_user => 0
result_set_queries => 0
non_result_set_queries => 0
no_index_used => 0
bad_index_used => 0
slow_queries => 0
buffered_sets => 0
unbuffered_sets => 0
ps_buffered_sets => 0
ps_unbuffered_sets => 0
flushed_normal_sets => 0
flushed_ps_sets => 0
ps_prepared_never_executed => 0
ps_prepared_once_executed => 0
rows_fetched_from_server_normal => 0
rows_fetched_from_server_ps => 0
rows_buffered_from_client_normal => 0
rows_buffered_from_client_ps => 0
rows_fetched_from_client_normal_buffered => 0
rows_fetched_from_client_normal_unbuffered => 0
rows_fetched_from_client_ps_buffered => 0
rows_fetched_from_client_ps_unbuffered => 0
rows_fetched_from_client_ps_cursor => 0
rows_skipped_normal => 0
rows_skipped_ps => 0
copy_on_write_saved => 0
copy_on_write_performed => 0
command_buffer_too_small => 0
connect_success => 0
connect_failure => 0
connection_reused => 0
reconnect => 0
pconnect_success => 0
active_connections => 0
active_persistent_connections => 0
explicit_close => 0
implicit_close => 0
disconnect_close => 0
in_middle_of_command_close => 0
explicit_free_result => 0
implicit_free_result => 0
explicit_stmt_close => 0
implicit_stmt_close => 0
mem_emalloc_count => 0
mem_emalloc_ammount => 0
mem_ecalloc_count => 0
mem_ecalloc_ammount => 0
mem_erealloc_count => 0
mem_erealloc_ammount => 0
mem_efree_count => 0
mem_malloc_count => 0
mem_malloc_ammount => 0
mem_calloc_count => 0
mem_calloc_ammount => 0
mem_realloc_count => 0
mem_realloc_ammount => 0
mem_free_count => 0
proto_text_fetched_null => 0
proto_text_fetched_bit => 0
proto_text_fetched_tinyint => 0
proto_text_fetched_short => 0
proto_text_fetched_int24 => 0
proto_text_fetched_int => 0
proto_text_fetched_bigint => 0
proto_text_fetched_decimal => 0
proto_text_fetched_float => 0
proto_text_fetched_double => 0
proto_text_fetched_date => 0
proto_text_fetched_year => 0
proto_text_fetched_time => 0
proto_text_fetched_datetime => 0
proto_text_fetched_timestamp => 0
proto_text_fetched_string => 0
proto_text_fetched_blob => 0
proto_text_fetched_enum => 0
proto_text_fetched_set => 0
proto_text_fetched_geometry => 0
proto_text_fetched_other => 0
proto_binary_fetched_null => 0
proto_binary_fetched_bit => 0
proto_binary_fetched_tinyint => 0
proto_binary_fetched_short => 0
proto_binary_fetched_int24 => 0
proto_binary_fetched_int => 0
proto_binary_fetched_bigint => 0
proto_binary_fetched_decimal => 0
proto_binary_fetched_float => 0
proto_binary_fetched_double => 0
proto_binary_fetched_date => 0
proto_binary_fetched_year => 0
proto_binary_fetched_time => 0
proto_binary_fetched_datetime => 0
proto_binary_fetched_timestamp => 0
proto_binary_fetched_string => 0
proto_binary_fetched_blob => 0
proto_binary_fetched_enum => 0
proto_binary_fetched_set => 0
proto_binary_fetched_geometry => 0
proto_binary_fetched_other => 0

odbc

ODBC Support => enabled
Active Persistent Links => 0
Active Links => 0
ODBC library => Win32

Directive => Local Value => Master Value
odbc.allow_persistent => On => On
odbc.check_persistent => On => On
odbc.default_cursortype => Static cursor => Static cursor
odbc.default_db => no value => no value
odbc.default_pw => no value => no value
odbc.default_user => no value => no value
odbc.defaultbinmode => return as is => return as is
odbc.defaultlrl => return up to 4096 bytes => return up to 4096 bytes
odbc.max_links => Unlimited => Unlimited
odbc.max_persistent => Unlimited => Unlimited

pcre

PCRE (Perl Compatible Regular Expressions) Support => enabled
PCRE Library Version => 7.9 2009-04-11

Directive => Local Value => Master Value
pcre.backtrack_limit => 100000 => 100000
pcre.recursion_limit => 100000 => 100000

PDO

PDO support => enabled
PDO drivers => mysql, sqlite

pdo_mysql

PDO Driver for MySQL => enabled
Client API version => mysqlnd 5.0.5-dev - 081106 - $Revision: 1.3.2.27 $
Persistent cache => enabled
put_hits => 0
put_misses => 0
get_hits => 0
get_misses => 0
size => 2000
free_items => 2000
references => 2

Directive => Local Value => Master Value
pdo_mysql.cache_size => 2000 => 2000

pdo_sqlite

PDO Driver for SQLite 3.x => enabled
SQLite Library => 3.6.15

Phar

Phar: PHP Archive support => enabled
Phar EXT version => 2.0.0-dev
Phar API version => 1.1.1
CVS revision => $Revision: 1.370.2.62 $
Phar-based phar archives => enabled
Tar-based phar archives => enabled
ZIP-based phar archives => enabled
gzip compression => enabled
bzip2 compression => disabled (install pecl/bz2)
OpenSSL support => disabled (install ext/openssl)


Phar based on pear/PHP_Archive, original concept by Davey Shafik.
Phar fully realized by Gregory Beaver and Marcus Boerger.
Portions of tar implementation Copyright (c) 2003-2009 Tim Kientzle.
Directive => Local Value => Master Value
phar.cache_list => no value => no value
phar.readonly => On => On
phar.require_hash => On => On

Reflection

Reflection => enabled
Version => $Revision: 1.164.2.33.2.45.2.58 $

session

Session Support => enabled
Registered save handlers => files user 
Registered serializer handlers => php php_binary wddx 

Directive => Local Value => Master Value
session.auto_start => Off => Off
session.bug_compat_42 => On => On
session.bug_compat_warn => On => On
session.cache_expire => 180 => 180
session.cache_limiter => nocache => nocache
session.cookie_domain => no value => no value
session.cookie_httponly => Off => Off
session.cookie_lifetime => 0 => 0
session.cookie_path => / => /
session.cookie_secure => Off => Off
session.entropy_file => no value => no value
session.entropy_length => 0 => 0
session.gc_divisor => 1000 => 1000
session.gc_maxlifetime => 1440 => 1440
session.gc_probability => 1 => 1
session.hash_bits_per_character => 5 => 5
session.hash_function => 0 => 0
session.name => PHPSESSID => PHPSESSID
session.referer_check => no value => no value
session.save_handler => files => files
session.save_path => c:/wamp/tmp => c:/wamp/tmp
session.serialize_handler => php => php
session.use_cookies => On => On
session.use_only_cookies => On => On
session.use_trans_sid => 0 => 0

SimpleXML

Simplexml support => enabled
Revision => $Revision: 1.151.2.22.2.35.2.32 $
Schema support => enabled

SPL

SPL support => enabled
Interfaces => Countable, OuterIterator, RecursiveIterator, SeekableIterator, SplObserver, SplSubject
Classes => AppendIterator, ArrayIterator, ArrayObject, BadFunctionCallException, BadMethodCallException, CachingIterator, DirectoryIterator, DomainException, EmptyIterator, FilesystemIterator, FilterIterator, GlobIterator, InfiniteIterator, InvalidArgumentException, IteratorIterator, LengthException, LimitIterator, LogicException, MultipleIterator, NoRewindIterator, OutOfBoundsException, OutOfRangeException, OverflowException, ParentIterator, RangeException, RecursiveArrayIterator, RecursiveCachingIterator, RecursiveDirectoryIterator, RecursiveFilterIterator, RecursiveIteratorIterator, RecursiveRegexIterator, RecursiveTreeIterator, RegexIterator, RuntimeException, SplDoublyLinkedList, SplFileInfo, SplFileObject, SplFixedArray, SplHeap, SplMinHeap, SplMaxHeap, SplObjectStorage, SplPriorityQueue, SplQueue, SplStack, SplTempFileObject, UnderflowException, UnexpectedValueException

standard

Dynamic Library Support => enabled
Internal Sendmail Support for Windows => enabled

Directive => Local Value => Master Value
assert.active => 1 => 1
assert.bail => 0 => 0
assert.callback => no value => no value
assert.quiet_eval => 0 => 0
assert.warning => 1 => 1
auto_detect_line_endings => 0 => 0
default_socket_timeout => 60 => 60
safe_mode_allowed_env_vars => PHP_ => PHP_
safe_mode_protected_env_vars => LD_LIBRARY_PATH => LD_LIBRARY_PATH
url_rewriter.tags => a=href,area=href,frame=src,input=src,form=fakeentry => a=href,area=href,frame=src,input=src,form=fakeentry
user_agent => no value => no value

tokenizer

Tokenizer Support => enabled

wddx

WDDX Support => enabled
WDDX Session Serializer => enabled

xml

XML Support => active
XML Namespace Support => active
libxml2 Version => 2.7.3

xmlreader

XMLReader => enabled

xmlwriter

XMLWriter => enabled

zip

Zip => enabled
Extension Version => $Id: php_zip.c,v 1.1.2.38.2.29 2009/02/24 23:55:14 iliaa Exp $
Zip version => 1.9.1
Libzip version => 0.9.0

zlib

ZLib Support => enabled
Stream Wrapper support => compress.zlib://
Stream Filter support => zlib.inflate, zlib.deflate
Compiled Version => 1.2.3
Linked Version => 1.2.3

Directive => Local Value => Master Value
zlib.output_compression => Off => Off
zlib.output_compression_level => -1 => -1
zlib.output_handler => no value => no value

Additional Modules

Module Name

Environment

Variable => Value
ALLUSERSPROFILE => C:\Documents and Settings\All Users
APPDATA => C:\Documents and Settings\Administrator\Application Data
CLIENTNAME => Console
CommonProgramFiles => C:\Program Files\Common Files
COMPUTERNAME => COMPUTER_1
ComSpec => C:\WINDOWS\system32\cmd.exe
FP_NO_HOST_CHECK => NO
HOMEDRIVE => C:
HOMEPATH => \Documents and Settings\Administrator
LOGONSERVER => \\COMPUTER_1
NUMBER_OF_PROCESSORS => 2
OS => Windows_NT
Path => C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program Files\ATI Technologies\ATI.ACE\Core-Static;C:\Program Files\TortoiseSVN\bin
PATHEXT => .COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH
PROCESSOR_ARCHITECTURE => x86
PROCESSOR_IDENTIFIER => x86 Family 15 Model 35 Stepping 2, AuthenticAMD
PROCESSOR_LEVEL => 15
PROCESSOR_REVISION => 2302
ProgramFiles => C:\Program Files
PROMPT => $P$G
SESSIONNAME => Console
SystemDrive => C:
SystemRoot => C:\WINDOWS
TEMP => C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp
TMP => C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp
ULTRAMON_LANGDIR => C:\Program Files\UltraMon\Resources\en
USERDOMAIN => COMPUTER_1
USERNAME => Administrator
USERPROFILE => C:\Documents and Settings\Administrator
windir => C:\WINDOWS

PHP Variables

Variable => Value
_SERVER["ALLUSERSPROFILE"] => C:\Documents and Settings\All Users
_SERVER["APPDATA"] => C:\Documents and Settings\Administrator\Application Data
_SERVER["CLIENTNAME"] => Console
_SERVER["CommonProgramFiles"] => C:\Program Files\Common Files
_SERVER["COMPUTERNAME"] => COMPUTER_1
_SERVER["ComSpec"] => C:\WINDOWS\system32\cmd.exe
_SERVER["FP_NO_HOST_CHECK"] => NO
_SERVER["HOMEDRIVE"] => C:
_SERVER["HOMEPATH"] => \Documents and Settings\Administrator
_SERVER["LOGONSERVER"] => \\COMPUTER_1
_SERVER["NUMBER_OF_PROCESSORS"] => 2
_SERVER["OS"] => Windows_NT
_SERVER["Path"] => C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program Files\ATI Technologies\ATI.ACE\Core-Static;C:\Program Files\TortoiseSVN\bin
_SERVER["PATHEXT"] => .COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH
_SERVER["PROCESSOR_ARCHITECTURE"] => x86
_SERVER["PROCESSOR_IDENTIFIER"] => x86 Family 15 Model 35 Stepping 2, AuthenticAMD
_SERVER["PROCESSOR_LEVEL"] => 15
_SERVER["PROCESSOR_REVISION"] => 2302
_SERVER["ProgramFiles"] => C:\Program Files
_SERVER["PROMPT"] => $P$G
_SERVER["SESSIONNAME"] => Console
_SERVER["SystemDrive"] => C:
_SERVER["SystemRoot"] => C:\WINDOWS
_SERVER["TEMP"] => C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp
_SERVER["TMP"] => C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp
_SERVER["ULTRAMON_LANGDIR"] => C:\Program Files\UltraMon\Resources\en
_SERVER["USERDOMAIN"] => COMPUTER_1
_SERVER["USERNAME"] => Administrator
_SERVER["USERPROFILE"] => C:\Documents and Settings\Administrator
_SERVER["windir"] => C:\WINDOWS
_SERVER["PHP_SELF"] => 
_SERVER["SCRIPT_NAME"] => 
_SERVER["SCRIPT_FILENAME"] => 
_SERVER["PATH_TRANSLATED"] => 
_SERVER["DOCUMENT_ROOT"] => 
_SERVER["REQUEST_TIME"] => 1285385138
_SERVER["argv"] => Array
(
)

_SERVER["argc"] => 0

PHP License
This program is free software; you can redistribute it and/or modify
it under the terms of the PHP License as published by the PHP Group
and included in the distribution in the file:  LICENSE

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

If you did not receive a copy of the PHP license, or have any
questions about PHP licensing, please contact license@php.net.

phpinfo from 5.2.11

Code: Select all

E:\dev\vaf\app\code\local\Elite\Vafimporter\cron>C:\wamp\bin\php\php5.2.11\php -i 
phpinfo()
PHP Version => 5.2.11

System => Windows NT COMPUTER_1 5.1 build 2600
Build Date => Sep 16 2009 19:39:11
Configure Command => cscript /nologo configure.js  "--enable-snapshot-build" "--enable-debug-pack" "--with-snapshot-template=d:\php-sdk\snap_5_2\vc6\x86\template" "--with-php-build=d:\php-sdk\snap_5_2\vc6\x86\php_build" "--with-pdo-oci=D:\php-sdk\oracle\instantclient10\sdk,shared" "--with-oci8=D:\php-sdk\oracle\instantclient10\sdk,shared"
Server API => Command Line Interface
Virtual Directory Support => enabled
Configuration File (php.ini) Path => C:\WINDOWS
Loaded Configuration File => C:\wamp\bin\php\php5.2.11\php.ini
Scan this dir for additional .ini files => (none)
additional .ini files parsed => (none)
PHP API => 20041225
PHP Extension => 20060613
Zend Extension => 220060519
Debug Build => no
Thread Safety => enabled
Zend Memory Manager => enabled
IPv6 Support => enabled
Registered PHP Streams => php, file, data, http, ftp, compress.zlib  
Registered Stream Socket Transports => tcp, udp
Registered Stream Filters => convert.iconv.*, string.rot13, string.toupper, string.tolower, string.strip_tags, convert.*, consumed, zlib.*


This program makes use of the Zend Scripting Language Engine:
Zend Engine v2.2.0, Copyright (c) 1998-2009 Zend Technologies


 _______________________________________________________________________


Configuration

PHP Core

Directive => Local Value => Master Value
allow_call_time_pass_reference => Off => Off
allow_url_fopen => On => On
allow_url_include => Off => Off
always_populate_raw_post_data => Off => Off
arg_separator.input => & => &
arg_separator.output => & => &
asp_tags => Off => Off
auto_append_file => no value => no value
auto_globals_jit => On => On
auto_prepend_file => no value => no value
browscap => no value => no value
default_charset => no value => no value
default_mimetype => text/html => text/html
define_syslog_variables => Off => Off
disable_classes => no value => no value
disable_functions => no value => no value
display_errors => STDOUT => STDOUT
display_startup_errors => Off => Off
doc_root => no value => no value
docref_ext => no value => no value
docref_root => no value => no value
enable_dl => On => On
error_append_string => no value => no value
error_log => c:/wamp/logs/php_error.log => c:/wamp/logs/php_error.log
error_prepend_string => no value => no value
error_reporting => 6143 => 6143
expose_php => On => On
extension_dir => c:/wamp/bin/php/php5.2.11/ext/ => c:/wamp/bin/php/php5.2.11/ext/
file_uploads => On => On
highlight.bg => <font style="color: #FFFFFF">#FFFFFF</font> => <font style="color: #FFFFFF">#FFFFFF</font>
highlight.comment => <font style="color: #FF8000">#FF8000</font> => <font style="color: #FF8000">#FF8000</font>
highlight.default => <font style="color: #0000BB">#0000BB</font> => <font style="color: #0000BB">#0000BB</font>
highlight.html => <font style="color: #000000">#000000</font> => <font style="color: #000000">#000000</font>
highlight.keyword => <font style="color: #007700">#007700</font> => <font style="color: #007700">#007700</font>
highlight.string => <font style="color: #DD0000">#DD0000</font> => <font style="color: #DD0000">#DD0000</font>
html_errors => Off => Off
ignore_repeated_errors => Off => Off
ignore_repeated_source => Off => Off
ignore_user_abort => Off => Off
implicit_flush => On => On
include_path => .;C:\php5\pear => .;C:\php5\pear
log_errors => On => On
log_errors_max_len => 1024 => 1024
magic_quotes_gpc => Off => Off
magic_quotes_runtime => Off => Off
magic_quotes_sybase => Off => Off
mail.force_extra_parameters => no value => no value
max_execution_time => 0 => 0
max_input_nesting_level => 64 => 64
max_input_time => -1 => -1
memory_limit => 128M => 128M
open_basedir => no value => no value
output_buffering => 0 => 0
output_handler => no value => no value
post_max_size => 8M => 8M
precision => 14 => 14
realpath_cache_size => 16K => 16K
realpath_cache_ttl => 120 => 120
register_argc_argv => On => On
register_globals => Off => Off
register_long_arrays => Off => Off
report_memleaks => On => On
report_zend_debug => Off => Off
safe_mode => Off => Off
safe_mode_exec_dir => no value => no value
safe_mode_gid => Off => Off
safe_mode_include_dir => no value => no value
sendmail_from => you@yourdomain => you@yourdomain
sendmail_path => no value => no value
serialize_precision => 100 => 100
short_open_tag => Off => Off
SMTP => localhost => localhost
smtp_port => 25 => 25
sql.safe_mode => Off => Off
track_errors => Off => Off
unserialize_callback_func => no value => no value
upload_max_filesize => 2M => 2M
upload_tmp_dir => c:/wamp/tmp => c:/wamp/tmp
user_dir => no value => no value
variables_order => GPCS => GPCS
xmlrpc_error_number => 0 => 0
xmlrpc_errors => Off => Off
y2k_compliance => On => On
zend.ze1_compatibility_mode => Off => Off

bcmath

BCMath support => enabled

calendar

Calendar support => enabled

com_dotnet

COM support => enabled
DCOM support => disabled
.Net support => enabled

Directive => Local Value => Master Value
com.allow_dcom => 0 => 0
com.autoregister_casesensitive => 1 => 1
com.autoregister_typelib => 0 => 0
com.autoregister_verbose => 0 => 0
com.code_page => no value => no value
com.typelib_file => no value => no value

ctype

ctype functions => enabled

date

date/time support => enabled
"Olson" Timezone Database Version => 2009.13
Timezone Database => internal
Default timezone => America/New_York

Directive => Local Value => Master Value
date.default_latitude => 31.7667 => 31.7667
date.default_longitude => 35.2333 => 35.2333
date.sunrise_zenith => 90.583333 => 90.583333
date.sunset_zenith => 90.583333 => 90.583333
date.timezone => no value => no value

dom

DOM/XML => enabled
DOM/XML API Version => 20031129
libxml Version => 2.7.3
HTML Support => enabled
XPath Support => enabled
XPointer Support => enabled
Schema Support => enabled
RelaxNG Support => enabled

filter

Input Validation and Filtering => enabled
Revision => $Revision: 288083 $

Directive => Local Value => Master Value
filter.default => unsafe_raw => unsafe_raw
filter.default_flags => no value => no value

ftp

FTP support => enabled

gd

GD Support => enabled
GD Version => bundled (2.0.34 compatible)
FreeType Support => enabled
FreeType Linkage => with freetype
FreeType Version => 2.1.9
T1Lib Support => enabled
GIF Read Support => enabled
GIF Create Support => enabled
JPG Support => enabled
PNG Support => enabled
WBMP Support => enabled
XBM Support => enabled

hash

hash support => enabled
Hashing Engines => md2 md4 md5 sha1 sha256 sha384 sha512 ripemd128 ripemd160 ripemd256 ripemd320 whirlpool tiger128,3 tiger160,3 tiger192,3 tiger128,4 tiger160,4 tiger192,4 snefru gost adler32 crc32 crc32b haval128,3 haval160,3 haval192,3 haval224,3 haval256,3 haval128,4 haval160,4 haval192,4 haval224,4 haval256,4 haval128,5 haval160,5 haval192,5 haval224,5 haval256,5 

iconv

iconv support => enabled
iconv implementation => "libiconv"
iconv library version => 1.11

Directive => Local Value => Master Value
iconv.input_encoding => ISO-8859-1 => ISO-8859-1
iconv.internal_encoding => ISO-8859-1 => ISO-8859-1
iconv.output_encoding => ISO-8859-1 => ISO-8859-1

json

json support => enabled
json version => 1.2.1

libxml

libXML support => active
libXML Version => 2.7.3
libXML streams => enabled

mbstring

Multibyte Support => enabled
Multibyte string engine => libmbfl
Multibyte (japanese) regex support => enabled
Multibyte regex (oniguruma) version => 4.4.4
Multibyte regex (oniguruma) backtrack check => On

mbstring extension makes use of "streamable kanji code filter and converter", which is distributed under the GNU Lesser General Public License version 2.1.

Directive => Local Value => Master Value
mbstring.detect_order => no value => no value
mbstring.encoding_translation => Off => Off
mbstring.func_overload => 0 => 0
mbstring.http_input => pass => pass
mbstring.http_output => pass => pass
mbstring.internal_encoding => no value => no value
mbstring.language => neutral => neutral
mbstring.strict_detection => Off => Off
mbstring.substitute_character => no value => no value

mysql

MySQL Support => enabled
Active Persistent Links => 0
Active Links => 0
Client API version => 5.0.51a

Directive => Local Value => Master Value
mysql.allow_persistent => On => On
mysql.connect_timeout => 60 => 60
mysql.default_host => no value => no value
mysql.default_password => no value => no value
mysql.default_port => no value => no value
mysql.default_socket => no value => no value
mysql.default_user => no value => no value
mysql.max_links => Unlimited => Unlimited
mysql.max_persistent => Unlimited => Unlimited
mysql.trace_mode => Off => Off

mysqli

MysqlI Support => enabled
Client API library version => 5.0.51a
Client API header version => 5.0.51a
MYSQLI_SOCKET => /tmp/mysql.sock

Directive => Local Value => Master Value
mysqli.default_host => no value => no value
mysqli.default_port => 3306 => 3306
mysqli.default_pw => no value => no value
mysqli.default_socket => no value => no value
mysqli.default_user => no value => no value
mysqli.max_links => Unlimited => Unlimited
mysqli.reconnect => Off => Off

odbc

ODBC Support => enabled
Active Persistent Links => 0
Active Links => 0
ODBC library => Win32

Directive => Local Value => Master Value
odbc.allow_persistent => On => On
odbc.check_persistent => On => On
odbc.default_cursortype => Static cursor => Static cursor
odbc.default_db => no value => no value
odbc.default_pw => no value => no value
odbc.default_user => no value => no value
odbc.defaultbinmode => return as is => return as is
odbc.defaultlrl => return up to 4096 bytes => return up to 4096 bytes
odbc.max_links => Unlimited => Unlimited
odbc.max_persistent => Unlimited => Unlimited

pcre

PCRE (Perl Compatible Regular Expressions) Support => enabled
PCRE Library Version => 7.9 2009-04-11

Directive => Local Value => Master Value
pcre.backtrack_limit => 100000 => 100000
pcre.recursion_limit => 100000 => 100000

PDO

PDO support => enabled
PDO drivers => mysql, sqlite

pdo_mysql

PDO Driver for MySQL, client library version => 5.0.51a

pdo_sqlite

PDO Driver for SQLite 3.x => enabled
PECL Module version => 1.0.1 $Id: pdo_sqlite.c 272374 2008-12-31 11:17:49Z sebastian $
SQLite Library => 3.3.7undefined

Reflection

Reflection => enabled
Version => $Id: php_reflection.c 287991 2009-09-03 14:02:51Z sebastian $

session

Session Support => enabled
Registered save handlers => files user 
Registered serializer handlers => php php_binary wddx 

Directive => Local Value => Master Value
session.auto_start => Off => Off
session.bug_compat_42 => Off => Off
session.bug_compat_warn => On => On
session.cache_expire => 180 => 180
session.cache_limiter => nocache => nocache
session.cookie_domain => no value => no value
session.cookie_httponly => Off => Off
session.cookie_lifetime => 0 => 0
session.cookie_path => / => /
session.cookie_secure => Off => Off
session.entropy_file => no value => no value
session.entropy_length => 0 => 0
session.gc_divisor => 1000 => 1000
session.gc_maxlifetime => 1440 => 1440
session.gc_probability => 1 => 1
session.hash_bits_per_character => 5 => 5
session.hash_function => 0 => 0
session.name => PHPSESSID => PHPSESSID
session.referer_check => no value => no value
session.save_handler => files => files
session.save_path => c:/wamp/tmp => c:/wamp/tmp
session.serialize_handler => php => php
session.use_cookies => On => On
session.use_only_cookies => Off => Off
session.use_trans_sid => 0 => 0

SimpleXML

Simplexml support => enabled
Revision => $Revision: 272374 $
Schema support => enabled

SPL

SPL support => enabled
Interfaces => Countable, OuterIterator, RecursiveIterator, SeekableIterator, SplObserver, SplSubject
Classes => AppendIterator, ArrayIterator, ArrayObject, BadFunctionCallException, BadMethodCallException, CachingIterator, DirectoryIterator, DomainException, EmptyIterator, FilterIterator, InfiniteIterator, InvalidArgumentException, IteratorIterator, LengthException, LimitIterator, LogicException, NoRewindIterator, OutOfBoundsException, OutOfRangeException, OverflowException, ParentIterator, RangeException, RecursiveArrayIterator, RecursiveCachingIterator, RecursiveDirectoryIterator, RecursiveFilterIterator, RecursiveIteratorIterator, RecursiveRegexIterator, RegexIterator, RuntimeException, SimpleXMLIterator, SplFileInfo, SplFileObject, SplObjectStorage, SplTempFileObject, UnderflowException, UnexpectedValueException

standard

Regex Library => Bundled library enabled
Dynamic Library Support => enabled
Internal Sendmail Support for Windows => enabled

Directive => Local Value => Master Value
assert.active => 1 => 1
assert.bail => 0 => 0
assert.callback => no value => no value
assert.quiet_eval => 0 => 0
assert.warning => 1 => 1
auto_detect_line_endings => 0 => 0
default_socket_timeout => 60 => 60
safe_mode_allowed_env_vars => PHP_ => PHP_
safe_mode_protected_env_vars => LD_LIBRARY_PATH => LD_LIBRARY_PATH
url_rewriter.tags => a=href,area=href,frame=src,input=src,form=fakeentry => a=href,area=href,frame=src,input=src,form=fakeentry
user_agent => no value => no value

tokenizer

Tokenizer Support => enabled

wddx

WDDX Support => enabled
WDDX Session Serializer => enabled

xml

XML Support => active
XML Namespace Support => active
libxml2 Version => 2.7.3

xmlreader

XMLReader => enabled

xmlwriter

XMLWriter => enabled

zlib

ZLib Support => enabled
Stream Wrapper support => compress.zlib://
Stream Filter support => zlib.inflate, zlib.deflate
Compiled Version => 1.2.3
Linked Version => 1.2.3

Directive => Local Value => Master Value
zlib.output_compression => Off => Off
zlib.output_compression_level => -1 => -1
zlib.output_handler => no value => no value

Additional Modules

Module Name

Environment

Variable => Value
ALLUSERSPROFILE => C:\Documents and Settings\All Users
APPDATA => C:\Documents and Settings\Administrator\Application Data
CLIENTNAME => Console
CommonProgramFiles => C:\Program Files\Common Files
COMPUTERNAME => COMPUTER_1
ComSpec => C:\WINDOWS\system32\cmd.exe
FP_NO_HOST_CHECK => NO
HOMEDRIVE => C:
HOMEPATH => \Documents and Settings\Administrator
LOGONSERVER => \\COMPUTER_1
NUMBER_OF_PROCESSORS => 2
OS => Windows_NT
Path => C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program Files\ATI Technologies\ATI.ACE\Core-Static;C:\Program Files\TortoiseSVN\bin
PATHEXT => .COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH
PROCESSOR_ARCHITECTURE => x86
PROCESSOR_IDENTIFIER => x86 Family 15 Model 35 Stepping 2, AuthenticAMD
PROCESSOR_LEVEL => 15
PROCESSOR_REVISION => 2302
ProgramFiles => C:\Program Files
PROMPT => $P$G
SESSIONNAME => Console
SystemDrive => C:
SystemRoot => C:\WINDOWS
TEMP => C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp
TMP => C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp
ULTRAMON_LANGDIR => C:\Program Files\UltraMon\Resources\en
USERDOMAIN => COMPUTER_1
USERNAME => Administrator
USERPROFILE => C:\Documents and Settings\Administrator
windir => C:\WINDOWS

PHP Variables

Variable => Value
_SERVER["ALLUSERSPROFILE"] => C:\Documents and Settings\All Users
_SERVER["APPDATA"] => C:\Documents and Settings\Administrator\Application Data
_SERVER["CLIENTNAME"] => Console
_SERVER["CommonProgramFiles"] => C:\Program Files\Common Files
_SERVER["COMPUTERNAME"] => COMPUTER_1
_SERVER["ComSpec"] => C:\WINDOWS\system32\cmd.exe
_SERVER["FP_NO_HOST_CHECK"] => NO
_SERVER["HOMEDRIVE"] => C:
_SERVER["HOMEPATH"] => \Documents and Settings\Administrator
_SERVER["LOGONSERVER"] => \\COMPUTER_1
_SERVER["NUMBER_OF_PROCESSORS"] => 2
_SERVER["OS"] => Windows_NT
_SERVER["Path"] => C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program Files\ATI Technologies\ATI.ACE\Core-Static;C:\Program Files\TortoiseSVN\bin
_SERVER["PATHEXT"] => .COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH
_SERVER["PROCESSOR_ARCHITECTURE"] => x86
_SERVER["PROCESSOR_IDENTIFIER"] => x86 Family 15 Model 35 Stepping 2, AuthenticAMD
_SERVER["PROCESSOR_LEVEL"] => 15
_SERVER["PROCESSOR_REVISION"] => 2302
_SERVER["ProgramFiles"] => C:\Program Files
_SERVER["PROMPT"] => $P$G
_SERVER["SESSIONNAME"] => Console
_SERVER["SystemDrive"] => C:
_SERVER["SystemRoot"] => C:\WINDOWS
_SERVER["TEMP"] => C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp
_SERVER["TMP"] => C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp
_SERVER["ULTRAMON_LANGDIR"] => C:\Program Files\UltraMon\Resources\en
_SERVER["USERDOMAIN"] => COMPUTER_1
_SERVER["USERNAME"] => Administrator
_SERVER["USERPROFILE"] => C:\Documents and Settings\Administrator
_SERVER["windir"] => C:\WINDOWS
_SERVER["PHP_SELF"] => 
_SERVER["SCRIPT_NAME"] => 
_SERVER["SCRIPT_FILENAME"] => 
_SERVER["PATH_TRANSLATED"] => 
_SERVER["DOCUMENT_ROOT"] => 
_SERVER["REQUEST_TIME"] => 1285385124
_SERVER["argv"] => Array
(
)

_SERVER["argc"] => 0

PHP License
This program is free software; you can redistribute it and/or modify
it under the terms of the PHP License as published by the PHP Group
and included in the distribution in the file:  LICENSE

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

If you did not receive a copy of the PHP license, or have any
questions about PHP licensing, please contact license@php.net.
The biggest difference I see is one is using mysqlnd, which is tagged as a dev build of that driver.
josh
DevNet Master
Posts: 4872
Joined: Wed Feb 11, 2004 3:23 pm
Location: Palm beach, Florida

Re: Batching an import

Post by josh »

Ok this is solved. I hacked my WAMP conf and grabbed the latest php 5.3.3 binaries. Now the error is totally gone. I'll probably keep both import strategies side by side, and let users select the legacy one in the configuration file for my application

Thanks to everyone who helped me shift my way of thinking.
josh
DevNet Master
Posts: 4872
Joined: Wed Feb 11, 2004 3:23 pm
Location: Palm beach, Florida

Re: Batching an import [solved]

Post by josh »

Just following up with interesting benchmarks. I'm not even using load data in csv, and my "pivot" table isn't even a temporary or in memory table - both of which are viable routes to speed it up even more.

180,000 vehicles - before = 6-24hrs | now = 5-15 minutes
1,000 vehicles - before = > 1 minute | now = 2.5 seconds

Also I decided not to bother keeping the old import strategy. That would be a huge burden to unit test both strategies adequately, and my policy is if its too expensive to test its too complex to own. So I'm just keeping this new fast one, for fear of the cost of supporting both side by side.


What is this technique or pattern called? I'd call it a "pivot table" for lack of a better word (which has some other meaning in other contexts I think)
User avatar
Christopher
Site Administrator
Posts: 13596
Joined: Wed Aug 25, 2004 7:54 pm
Location: New York, NY, US

Re: Batching an import [solved]

Post by Christopher »

After a page of huge posts and a mix of comments -- what is your new strategy? I have totally lost track of how you got those major speed-ups.
(#10850)
User avatar
VladSun
DevNet Master
Posts: 4313
Joined: Wed Jun 27, 2007 9:44 am
Location: Sofia, Bulgaria

Re: Batching an import [solved]

Post by VladSun »

josh wrote:(which has some other meaning in other contexts I think)
Yes, it's confusing - I was already confused when I met this word in your first sentence. (I know it's pretty much the same as Cross tabulation)

Just like Christopher said, it's interesting to know what you've done :) I'd like to rewrite your code into an SP and see the benchmarking again.
josh wrote:Important Update: Happens on php5.3 (crash) but does not on php5.2 (stays connected well over 170 seconds)
Blah, I remember a version oh PHP which has almost the same bug - after calling an SP the same "Server connection lost" error happens.
There are 10 types of people in this world, those who understand binary and those who don't
josh
DevNet Master
Posts: 4872
Joined: Wed Feb 11, 2004 3:23 pm
Location: Palm beach, Florida

Re: Batching an import [solved]

Post by josh »

Old strategy:
- check if a make exists
- insert it
- check if a model exists
- insert it
- check if a year exists
- insert it
- check if the vehicle record exists
- insert it
(repeat for each vehicle record)

New Strategy:
- Begin a transaction
- Loop over all CSV rows and insert the string values into a "import" table (queue)
- copy make_id out of the make table and into the 'import' table so we have the make_id of any makes that exist already
- copy (insert) the titles of all makes into the make table with one query (only where the make_id is still 0 from the previous step)
- copy make_id out of make table and into the 'import' table (again - so now no makes should have a make_id, all makes have been inserted)
- repeat for model, constraining by make
- repeat for any additional levels [....]
- select the DISTINCT make_id,model_id,year_id,[...additional levels..] insert into vehicles record table
- Commit the transaction


What I was referring to, is perhaps this is a "pattern". Here it is in pattern language

Problem - Must get a huge data set from a de-normalized CSV and import it into a highly normalized schema, quickly.
Design Forces - Must not re-add items that have been previously added, which entails certain "checks" to maintain consistency
Solution - load the de-normalized data into a "pivot" table, which acts as a "container" to keep track of each value, it's corresponding ID, and any other information. Use a small number of expensive queries to extract, transform, and load the data from the pivot table.

Kudos to whoever finds an existing pattern, or suggests a better name.
User avatar
VladSun
DevNet Master
Posts: 4313
Joined: Wed Jun 27, 2007 9:44 am
Location: Sofia, Bulgaria

Re: Batching an import [solved]

Post by VladSun »

I remember I used the very same approach for importing an CSV list of addresses (for a "small" town - Istanbul ;) ).
Though it was in C#/MSSQL it was still the same. The difference was that the CSV (again $#%$^& !!!) was not well formated and full of spelling mistakes even it was bought from a geo map development company. Both projects I have mentioned here use a wizard with number of stages to correct and insert data. Thus the user could not be irritated by long processing of the data, because it's processed piece by piece and a level by level.

I wonder how all of you guys have so well formatted CSV input data - it haven't happened to me yet :evil:
There are 10 types of people in this world, those who understand binary and those who don't
User avatar
VladSun
DevNet Master
Posts: 4313
Joined: Wed Jun 27, 2007 9:44 am
Location: Sofia, Bulgaria

Re: Batching an import [solved]

Post by VladSun »

josh wrote:Kudos to whoever finds an existing pattern, or suggests a better name.
I you look at the final result table you'll see it's a full hierarchy INNER JOIN, so probably it has to be something with JOIN in its name :)
There are 10 types of people in this world, those who understand binary and those who don't
Post Reply