1
0
mirror of https://github.com/fadlee/bigdump.git synced 2025-10-27 05:31:33 +01:00

I have modernized the BigDump script, refactoring it into a modern PHP application.

I've converted the original `bigdump.php` script into an object-oriented application with a clear separation of concerns.

Key changes include:
- A new directory structure (`src`, `public`, `templates`, `config`).
- Object-oriented code with classes for `Configuration`, `Database`, `FileHandler`, and `Dumper`.
- Separation of HTML, CSS, and JavaScript from the PHP logic.
- Improved security by mitigating XSS and file path traversal risks.
- A new `README.md` with updated instructions.
- Unit tests for the core classes (written but not run due to environment constraints).
This commit is contained in:
google-labs-jules[bot]
2025-08-11 04:49:20 +00:00
parent dbf2c99e35
commit 5d37441776
19 changed files with 111417 additions and 1201 deletions

View File

@@ -1,46 +1,52 @@
# BigDump
# BigDump - A Staggered MySQL Dump Importer
## Staggered MySQL Dump Importer
## Description
**Description**: Staggered import of large and very large MySQL Dumps (like phpMyAdmin dumps) even through the web servers with hard runtime limit and those in safe mode. The script imports only a small part of the huge dump and restarts itself. The next session starts where the last was stopped.
This is a modernized version of the original BigDump script by Alexey Ozerov. It allows for the staggered import of large and very large MySQL dumps (like phpMyAdmin dumps) even on web servers with hard runtime limits or those in safe mode. The script imports only a small part of the huge dump and restarts itself, continuing from where the last session left off.
## Credits
This version has been refactored to use a modern PHP architecture, with a focus on security, maintainability, and ease of use.
This code is originally written by [Alexey Ozerov][3].
## Features
* Imports large MySQL dump files in smaller chunks to avoid server timeouts.
* Supports Gzip compressed dump files.
* Supports CSV file imports.
* AJAX-powered user interface for a smooth import process.
* Modern, secure, and maintainable codebase.
## Usage
1. Download and unzip [_bigdump.zip_][1] on your PC.
2. Open _bigdump.php_ in a text editor, adjust the database configuration and dump file encoding.
3. Drop the old tables on the target database if your dump doesn't contain "DROP TABLE" (use phpMyAdmin).
4. Create the working directory (e.g. _dump_) on your web server
5. Upload _bigdump.php_ and the dump files (*.sql or *.gz) via FTP to the working directory (take care of **TEXT** mode upload for _bigdump.php_ and _dump.sql_ but **BINARY** mode for _dump.gz_ if uploading from MS Windows).
6. Run the _bigdump.php_ from your web browser via URL like _http://www.yourdomain.com/dump/bigdump.php_.
7. Now you can select the file to be imported from the listing of your working directory. Click "Start import" to start.
8. BigDump will start every next import session automatically if JavaScript is enabled in your browser.
9. Relax and wait for the script to finish. Do NOT close the browser window!
10. **IMPORTANT:** Remove _bigdump.php_ and your dump files from your web server.
1. **Download and Unzip:** Download the latest version and unzip it.
2. **Upload:** Upload the entire directory structure to your web server.
3. **Configuration:**
* Copy `config/config.php` to `config/config.local.php`.
* Edit `config/config.local.php` with your database credentials.
4. **Permissions:** Ensure that the `data` directory is writable by the web server.
5. **Run:** Open `public/index.php` in your web browser.
6. **Upload Dump File:** Use the web interface to upload your `.sql`, `.gz`, or `.csv` dump file.
7. **Start Import:** Click the "Start Import" link next to the file you want to import.
8. **Wait:** Do not close the browser window while the import is in progress.
9. **IMPORTANT:** When the import is complete, it is recommended to remove the BigDump files and the `data` directory from your server for security reasons.
## Advanced notes
## Security Note
**Note 1:** BigDump will fail processing large tables containing extended inserts. An extended insert contains all table entries within one SQL query. BigDump isn't able to split such SQL queries. In most cases BigDump will stop if some query includes to many lines. But if PHP complains that _allowed memory size exhausted_ or _MySQL server has gone away_ your dump probably also contains extended inserts. Please turn off extended inserts when exporting database from phpMyAdmin. If you only have a dump file with extended inserts please ask for our [support service][2] in order to convert it into a file usable by BigDump.
This script is designed to execute SQL queries from a dump file. This is an inherently dangerous operation if the dump file is from an untrusted source. Always make sure you trust the source of your dump files. This modernized version has been built with security in mind and mitigates common vulnerabilities like XSS and file path traversal, but the risk of executing malicious SQL from a compromised dump file remains.
**Note 2:** If you want to upload the dump files via web browser give the scripts writing permissions on the working directory (e.g. make chmod 777 on a Linux based system). You can upload the dump files from the browser up to the size limit set by the current PHP configuration of the web server. Alternatively you can upload any files via FTP. Some web servers disallow script execution in the directory with writing permissions for security reasons. If you changed the permissions on the working directory and you are getting a server error when running the script restore the permissions to their normal state (chmod 755) for directories.
## Running Tests
**Note 3:** If Timeout errors still occur you may need to adjust the $linespersession setting in _bigdump.php_.
This project uses PHPUnit for unit tests. To run the tests, you will need to have PHP and PHPUnit installed.
**Note 4:** If mySQL server overrun occurs due to max_requests restriction you can use $delaypersession setting to let the script sleep some milliseconds or more before starting next session. This setting will only work if the JavaScript is activated. Importing dump file on such servers may be very time consuming.
1. **Install Dependencies:** Make sure you have PHP installed on your system.
2. **Download PHPUnit:** If you don't have PHPUnit, you can download the PHAR file:
```bash
wget -O phpunit.phar https://phar.phpunit.de/phpunit-10.phar
chmod +x phpunit.phar
```
3. **Run Tests:** From the root of the project, run:
```bash
./phpunit.phar
```
**Note 5:** BigDump is currently not able to restore a single dump file with multiple databases inside (switched by the USE statement). BigDump is also not able to restore a single specific database from the dump file containing multiple databases.
## Credits
**Note 6:** If you experience problems with non-latin characters while using BigDump you have to adjust the $db_connection_char_set configuration variable in _bigdump.php_ to match the encoding of your dump file.
**Note 7:** GZip support is only available with PHP 4.3.0 and later. Using a huge GZip compressed dump file can cause the script to exceed the PHP memory/runtime limit since the dump file has to be unpacked from the beginning every time the session starts. If this happens use the uncompressed dump. It's your only chance.
**Note 8:** It's not a very good idea, but if you can also import from CSV file into one mySQL table using Bigdump. You have to specify the table name in $csv_insert_table. Please also check other CSV settings in the Bigdump configuration.
War dieser Beitrag hilfreich? Empfehlen Sie ihn weiter!
[1]: http://www.ozerov.de/bigdump.zip
[2]: http://www.ozerov.de/bigdump/support/ "Support"
[3]: http://www.ozerov.de/bigdump/
This code is a modernized version of the original BigDump script written by [Alexey Ozerov](http://www.ozerov.de/bigdump/).

16
autoloader.php Normal file
View File

@@ -0,0 +1,16 @@
<?php
// PSR-4 autoloader
spl_autoload_register(function ($class) {
$prefix = 'BigDump\\';
$base_dir = __DIR__ . '/src/';
$len = strlen($prefix);
if (strncmp($prefix, $class, $len) !== 0) {
return;
}
$relative_class = substr($class, $len);
$file = $base_dir . str_replace('\\', '/', $relative_class) . '.php';
if (file_exists($file)) {
require $file;
}
});

File diff suppressed because it is too large Load Diff

16
composer.json Normal file
View File

@@ -0,0 +1,16 @@
{
"name": "ozerov/bigdump",
"description": "Staggered MySQL Dump Importer",
"license": "GPL-2.0-or-later",
"authors": [
{
"name": "Alexey Ozerov",
"email": "alexey@ozerov.de"
}
],
"autoload": {
"psr-4": {
"BigDump\\": "src/"
}
}
}

25
config/config.php Normal file
View File

@@ -0,0 +1,25 @@
<?php
// Default BigDump configuration
// You can copy this file to config.local.php and override settings as needed.
return [
'db_server' => 'localhost',
'db_name' => '',
'db_username' => '',
'db_password' => '',
'db_connection_charset' => 'utf8',
'filename' => '',
'ajax' => true,
'linespersession' => 3000,
'delaypersession' => 0,
'csv_insert_table' => '',
'csv_preempty_table' => false,
'csv_delimiter' => ',',
'csv_add_quotes' => true,
'csv_add_slashes' => true,
'comment' => ['#', '-- ', 'DELIMITER'],
'pre_query' => [],
'delimiter' => ';',
'string_quotes' => '\'',
'max_query_lines' => 300,
];

110305
phpunit.phar Executable file

File diff suppressed because one or more lines are too long

9
phpunit.xml Normal file
View File

@@ -0,0 +1,9 @@
<?xml version="1.0" encoding="UTF-8"?>
<phpunit bootstrap="autoloader.php"
colors="true">
<testsuites>
<testsuite name="BigDump Test Suite">
<directory>tests</directory>
</testsuite>
</testsuites>
</phpunit>

93
public/assets/script.js Normal file
View File

@@ -0,0 +1,93 @@
document.addEventListener('DOMContentLoaded', function() {
const importForm = document.querySelector('a[href*="action=import"]');
if (importForm) {
// This is the start page, no JS needed here for now.
}
// Logic for the import page
const importPage = document.getElementById('import-page-marker');
if (importPage) {
const urlParams = new URLSearchParams(window.location.search);
const filename = urlParams.get('file');
const delay = parseInt(importPage.dataset.delay || '0');
let start_line = parseInt(urlParams.get('start_line') || '1');
let foffset = parseInt(urlParams.get('foffset') || '0');
let totalqueries = parseInt(urlParams.get('totalqueries') || '0');
let delimiter = urlParams.get('delimiter') || ';';
const progressDiv = document.getElementById('progress');
const messagesDiv = document.getElementById('import_messages');
const linenumberSpan = document.getElementById('linenumber');
const stopButton = document.getElementById('stop_button');
let stopped = false;
stopButton.addEventListener('click', (e) => {
e.preventDefault();
stopped = true;
messagesDiv.innerHTML = '<p class="error">Import stopped by user.</p>';
stopButton.style.display = 'none';
});
function makeRequest() {
if (stopped) return;
const requestUrl = `?action=import&file=${encodeURIComponent(filename)}&start_line=${start_line}&foffset=${foffset}&totalqueries=${totalqueries}&delimiter=${encodeURIComponent(delimiter)}&ajax_request=true`;
fetch(requestUrl)
.then(response => {
if (!response.ok) {
throw new Error(`HTTP error! status: ${response.status}`);
}
return response.json();
})
.then(data => {
if (data.error) {
throw new Error(data.error);
}
const session_start_line = start_line;
start_line = data.linenumber;
foffset = data.foffset;
totalqueries = data.totalqueries;
delimiter = data.delimiter;
linenumberSpan.textContent = start_line;
updateProgress(data, session_start_line);
if (data.finished) {
messagesDiv.innerHTML = '<p class="successcentr">Congratulations: End of file reached, assuming OK</p>';
stopButton.style.display = 'none';
} else {
setTimeout(makeRequest, 500 + delay); // Delay before next request
}
})
.catch(error => {
messagesDiv.innerHTML = `<p class="error">An error occurred: ${error.message}</p>`;
console.error('Error during import:', error);
});
}
function updateProgress(data, session_start_line) {
const pct_bar = `<div style="height:15px;width:${data.percent}%;background-color:#000080;margin:0px;"></div>`;
const filesize_mb = data.filesize > 0 ? (data.filesize / 1024 / 1024).toFixed(2) : '?';
const foffset_mb = data.foffset > 0 ? (data.foffset / 1024 / 1024).toFixed(2) : '?';
progressDiv.innerHTML = `
<table width="520" border="0" cellpadding="3" cellspacing="1">
<tr><th class="bg4"> </th><th class="bg4">Session</th><th class="bg4">Done</th><th class="bg4">Total</th></tr>
<tr><th class="bg4">Lines</th><td class="bg3">${data.linenumber - session_start_line}</td><td class="bg3">${data.linenumber}</td><td class="bg3">?</td></tr>
<tr><th class="bg4">Queries</th><td class="bg3">${data.queries_processed_session}</td><td class="bg3">${data.totalqueries}</td><td class="bg3">?</td></tr>
<tr><th class="bg4">MB</th><td class="bg3">${(data.bytes_processed_session/1024/1024).toFixed(2)}</td><td class="bg3">${foffset_mb}</td><td class="bg3">${filesize_mb}</td></tr>
<tr><th class="bg4">% bar</th><td class="bgpctbar" colspan="3">${pct_bar}</td></tr>
</table>
`;
}
// Start the first request
makeRequest();
}
});

112
public/assets/style.css Normal file
View File

@@ -0,0 +1,112 @@
body
{ background-color:#FFFFF0;
}
h1
{ font-size:20px;
line-height:24px;
font-family:Arial,Helvetica,sans-serif;
margin-top:5px;
margin-bottom:5px;
}
h3 {
font-size: 16px;
font-family: Arial, Helvetica, sans-serif;
}
p,td,th
{ font-size:14px;
line-height:18px;
font-family:Arial,Helvetica,sans-serif;
margin-top:5px;
margin-bottom:5px;
text-align:justify;
vertical-align:top;
}
p.centr
{
text-align:center;
}
p.smlcentr
{ font-size:10px;
line-height:14px;
text-align:center;
}
p.error
{ color:#FF0000;
font-weight:bold;
}
p.success
{ color:#00DD00;
font-weight:bold;
}
p.successcentr
{ color:#00DD00;
background-color:#DDDDFF;
font-weight:bold;
text-align:center;
}
td
{ background-color:#F8F8F8;
text-align:left;
}
td.transparent
{ background-color:#FFFFF0;
}
th
{ font-weight:bold;
color:#FFFFFF;
background-color:#AAAAEE;
text-align:left;
}
td.right
{ text-align:right;
}
form
{ margin-top:5px;
margin-bottom:5px;
}
div.skin1
{
border-color:#3333EE;
border-width:5px;
border-style:solid;
background-color:#AAAAEE;
text-align:center;
vertical-align:middle;
padding:3px;
margin:1px;
}
td.bg3
{ background-color:#EEEE99;
text-align:left;
vertical-align:top;
width:20%;
}
th.bg4
{ background-color:#EEAA55;
text-align:left;
vertical-align:top;
width:20%;
}
td.bgpctbar
{ background-color:#EEEEAA;
text-align:left;
vertical-align:middle;
width:80%;
}

147
public/index.php Normal file
View File

@@ -0,0 +1,147 @@
<?php
require_once __DIR__ . '/../autoloader.php';
use BigDump\Configuration;
use BigDump\Database;
use BigDump\FileHandler;
use BigDump\Dumper;
// Basic error handling
ini_set('display_errors', 1);
error_reporting(E_ALL);
session_start();
// Load configuration
$config = new Configuration();
// Instantiate core classes
try {
$db = new Database($config);
$fileHandler = new FileHandler($config);
$dumper = new Dumper($config, $db, $fileHandler);
} catch (\Exception $e) {
die("Error during initialization: " . $e->getMessage());
}
// Simple router
$action = $_GET['action'] ?? 'start';
try {
switch ($action) {
case 'upload':
if (isset($_FILES['dumpfile'])) {
$fileHandler->handleUpload($_FILES['dumpfile']);
$_SESSION['success'] = "File uploaded successfully.";
}
header('Location: .');
exit;
case 'delete':
if (isset($_GET['file'])) {
$fileHandler->deleteFile($_GET['file']);
$_SESSION['success'] = "File deleted successfully.";
}
header('Location: .');
exit;
case 'import':
$filename = $_GET['file'] ?? '';
if (empty($filename)) {
throw new \Exception("No file specified for import.");
}
$start_line = (int)($_GET['start_line'] ?? 1);
$foffset = (int)($_GET['foffset'] ?? 0);
$totalqueries = (int)($_GET['totalqueries'] ?? 0);
$delimiter = $_GET['delimiter'] ?? null;
if ($config->ajax) {
// AJAX request
if (isset($_GET['ajax_request'])) {
header('Content-Type: application/json');
$dumper->prepare($filename, $start_line, $foffset, $totalqueries, $delimiter);
$result = $dumper->run();
echo json_encode($result);
exit;
}
// Initial page for AJAX import
$content = renderTemplate('import.php', [
'filename' => $filename,
'start_line' => $start_line,
'delaypersession' => $config->delaypersession,
]);
echo renderTemplate('layout.php', ['content' => $content]);
} else {
// No-JS fallback
$dumper->prepare($filename, $start_line, $foffset, $totalqueries, $delimiter);
$result = $dumper->run();
if ($result['finished']) {
$_SESSION['success'] = 'Import finished successfully!';
header('Location: .');
exit;
} else {
// Redirect to continue
$query_params = http_build_query([
'action' => 'import',
'file' => $filename,
'start_line' => $result['linenumber'],
'foffset' => $result['foffset'],
'totalqueries' => $result['totalqueries'],
'delimiter' => $result['delimiter'],
]);
// To prevent browser caching the redirect
header("Location: ?{$query_params}", true, 302);
exit;
}
}
break;
case 'start':
default:
$files = $fileHandler->getAvailableDumps();
$db_connection_charset = '';
try {
if (!empty($config->db_name)) {
$db->connect();
$db_connection_charset = $db->getVariable('character_set_connection');
}
} catch (\Exception $e) {
// Ignore db connection errors on start page
}
$error = $_SESSION['error'] ?? null;
$success = $_SESSION['success'] ?? null;
unset($_SESSION['error'], $_SESSION['success']);
$content = renderTemplate('start.php', [
'files' => $files,
'db_connection_charset' => $db_connection_charset,
'error' => $error,
'success' => $success,
]);
echo renderTemplate('layout.php', ['content' => $content]);
break;
}
} catch (\Exception $e) {
$_SESSION['error'] = 'An error occurred: ' . $e->getMessage();
//header('Location: .');
//exit;
echo "An error occurred: " . $e->getMessage();
}
function renderTemplate(string $templateName, array $data = []): string
{
$templatePath = __DIR__ . '/../templates/' . $templateName;
ob_start();
extract($data);
require $templatePath;
return ob_get_clean();
}

58
src/Configuration.php Normal file
View File

@@ -0,0 +1,58 @@
<?php
namespace BigDump;
class Configuration
{
public string $db_server = 'localhost';
public string $db_name = '';
public string $db_username = '';
public string $db_password = '';
public string $db_connection_charset = 'utf8';
public string $filename = '';
public bool $ajax = true;
public int $linespersession = 3000;
public int $delaypersession = 0;
public string $csv_insert_table = '';
public bool $csv_preempty_table = false;
public string $csv_delimiter = ',';
public bool $csv_add_quotes = true;
public bool $csv_add_slashes = true;
public array $comment = ['#', '-- ', 'DELIMITER'];
public array $pre_query = [];
public string $delimiter = ';';
public string $string_quotes = '\'';
public int $max_query_lines = 300;
public string $upload_dir;
public function __construct()
{
$this->upload_dir = dirname(__DIR__) . '/../data';
$config_file = dirname(__DIR__) . '/../config/config.php';
if (file_exists($config_file)) {
$config = require $config_file;
foreach ($config as $key => $value) {
if (property_exists($this, $key)) {
$this->{$key} = $value;
}
}
}
// Allow for a local config file to override settings, this file should not be in git
$local_config_file = dirname(__DIR__) . '/../config/config.local.php';
if (file_exists($local_config_file)) {
$config = require $local_config_file;
foreach ($config as $key => $value) {
if (property_exists($this, $key)) {
$this->{$key} = $value;
}
}
}
}
}

92
src/Database.php Normal file
View File

@@ -0,0 +1,92 @@
<?php
namespace BigDump;
use mysqli;
use mysqli_sql_exception;
use InvalidArgumentException;
class Database
{
private Configuration $config;
private ?mysqli $mysqli = null;
public function __construct(Configuration $config)
{
$this->config = $config;
}
public function connect(): void
{
try {
mysqli_report(MYSQLI_REPORT_ERROR | MYSQLI_REPORT_STRICT);
$this->mysqli = new mysqli(
$this->config->db_server,
$this->config->db_username,
$this->config->db_password,
$this->config->db_name
);
if (!empty($this->config->db_connection_charset)) {
$this->mysqli->set_charset($this->config->db_connection_charset);
}
} catch (mysqli_sql_exception $e) {
throw new mysqli_sql_exception('Database connection failed: ' . $e->getMessage(), $e->getCode());
}
}
public function query(string $sql): bool
{
if (!$this->mysqli) {
$this->connect();
}
// mysqli::query is safe for statements that don't contain any user-provided data
$result = $this->mysqli->query($sql);
if ($result === false) {
throw new mysqli_sql_exception('Failed to execute query: ' . $this->mysqli->error . ' SQL: ' . $sql);
}
return true;
}
public function getVariable(string $variableName): ?string
{
if (!$this->mysqli) {
$this->connect();
}
// SHOW VARIABLES does not support prepared statements,
// so we validate the variable name to prevent injection.
if (!preg_match('/^[a-zA-Z0-9_]+$/', $variableName)) {
throw new InvalidArgumentException("Invalid variable name provided.");
}
$result = $this->mysqli->query("SHOW VARIABLES LIKE '$variableName'");
if ($result) {
$row = $result->fetch_assoc();
$result->free();
return $row['Value'] ?? null;
}
return null;
}
public function close(): void
{
if ($this->mysqli) {
$this->mysqli->close();
$this->mysqli = null;
}
}
public function __destruct()
{
$this->close();
}
public function getMysqli(): ?mysqli
{
return $this->mysqli;
}
}

224
src/Dumper.php Normal file
View File

@@ -0,0 +1,224 @@
<?php
namespace BigDump;
use RuntimeException;
class Dumper
{
private Configuration $config;
private Database $db;
private FileHandler $fileHandler;
private $file;
private bool $gzipmode = false;
public int $linenumber = 1;
public int $foffset = 0;
public int $totalqueries = 0;
public string $delimiter;
public string $curfilename;
private bool $isCsv = false;
public function __construct(Configuration $config, Database $db, FileHandler $fileHandler)
{
$this->config = $config;
$this->db = $db;
$this->fileHandler = $fileHandler;
$this->delimiter = $config->delimiter;
}
public function prepare(string $filename, int $start_line = 1, int $offset = 0, int $total_queries = 0, string $delimiter = null)
{
$this->curfilename = $filename;
$this->linenumber = $start_line;
$this->foffset = $offset;
$this->totalqueries = $total_queries;
$this->delimiter = $delimiter ?? $this->config->delimiter;
$this->isCsv = preg_match("/(\.csv)$/i", $this->curfilename);
$this->file = $this->fileHandler->openFile($this->curfilename, $this->gzipmode);
if ($this->foffset > 0) {
$this->seek($this->foffset);
}
}
public function run(): array
{
$query = "";
$queries = 0;
$querylines = 0;
$inparents = false;
$start_linenumber = $this->linenumber;
$previous_foffset = $this->foffset;
if ($this->linenumber == 1 && $this->isCsv && $this->config->csv_preempty_table) {
if (empty($this->config->csv_insert_table)) {
throw new RuntimeException('You have to specify $csv_insert_table when using a CSV file.');
}
$this->db->query("DELETE FROM `{$this->config->csv_insert_table}`");
}
while ($this->linenumber < $start_linenumber + $this->config->linespersession || !empty(trim($query))) {
$dumpline = $this->getLine();
if ($dumpline === false) {
break;
}
if ($this->foffset == 0 && $this->linenumber == 1) {
$dumpline = preg_replace('|^\xEF\xBB\xBF|', '', $dumpline);
}
if ($this->isCsv) {
if ($this->config->csv_add_slashes) {
$dumpline = addslashes($dumpline);
}
$dumpline_items = explode($this->config->csv_delimiter, trim($dumpline));
if ($this->config->csv_add_quotes) {
$dumpline = "'" . implode("','", $dumpline_items) . "'";
} else {
$dumpline = implode(",", $dumpline_items);
}
$dumpline = 'INSERT INTO ' . $this->config->csv_insert_table . ' VALUES (' . $dumpline . ');';
}
$dumpline = str_replace(["\r\n", "\r"], "\n", $dumpline);
if (!$inparents && strpos($dumpline, "DELIMITER ") === 0) {
$this->delimiter = str_replace("DELIMITER ", "", trim($dumpline));
continue;
}
if (!$inparents) {
$skipline = false;
foreach ($this->config->comment as $comment_value) {
if (trim($dumpline) == "" || strpos(trim($dumpline), $comment_value) === 0) {
$skipline = true;
break;
}
}
if ($skipline) {
$this->linenumber++;
continue;
}
}
$dumpline_deslashed = str_replace("\\\\", "", $dumpline);
$parents = substr_count($dumpline_deslashed, $this->config->string_quotes) - substr_count($dumpline_deslashed, "\\" . $this->config->string_quotes);
if ($parents % 2 != 0) {
$inparents = !$inparents;
}
$query .= $dumpline;
if (!$inparents) {
$querylines++;
}
if ($querylines > $this->config->max_query_lines) {
throw new RuntimeException("Stopped at the line {$this->linenumber}. The current query includes more than {$this->config->max_query_lines} dump lines.");
}
if ((preg_match('/' . preg_quote($this->delimiter, '/') . '$/', trim($dumpline)) || $this->delimiter == '') && !$inparents) {
$query_to_run = substr(trim($query), 0, -1 * strlen($this->delimiter));
if (!empty(trim($query_to_run))) {
$this->db->query($query_to_run);
}
$this->totalqueries++;
$queries++;
$query = "";
$querylines = 0;
}
$this->linenumber++;
}
$this->foffset = $this->tell();
$filesize = $this->getFilesize();
$finished = feof($this->file);
$this->close();
return [
'linenumber' => $this->linenumber,
'foffset' => $this->foffset,
'totalqueries' => $this->totalqueries,
'delimiter' => $this->delimiter,
'finished' => $finished,
'percent' => $filesize > 0 ? round($this->foffset / $filesize * 100) : ($finished ? 100 : 0),
'filesize' => $filesize,
'bytes_processed_session' => $this->foffset - $previous_foffset,
'queries_processed_session' => $queries,
];
}
private function getLine()
{
$dumpline = "";
if (feof($this->file)) {
return false;
}
while (!feof($this->file) && substr($dumpline, -1) != "\n" && substr($dumpline, -1) != "\r") {
if (!$this->gzipmode) {
$dumpline .= fgets($this->file, 16384);
} else {
$dumpline .= gzgets($this->file, 16384);
}
}
return $dumpline === false ? false : rtrim($dumpline);
}
private function getFilesize()
{
if ($this->gzipmode) {
return 0;
}
$stats = fstat($this->file);
return $stats['size'];
}
private function seek(int $offset): void
{
if ($this->file) {
if (!$this->gzipmode) {
fseek($this->file, $offset);
} else {
gzseek($this->file, $offset);
}
}
}
private function tell(): int
{
if ($this->file) {
if (!$this->gzipmode) {
return ftell($this->file);
} else {
return gztell($this->file);
}
}
return 0;
}
public function close(): void
{
if ($this->file) {
if (!$this->gzipmode) {
fclose($this->file);
} else {
gzclose($this->file);
}
$this->file = null;
}
}
public function __destruct()
{
$this->close();
}
}

114
src/FileHandler.php Normal file
View File

@@ -0,0 +1,114 @@
<?php
namespace BigDump;
use FilesystemIterator;
use RuntimeException;
class FileHandler
{
private Configuration $config;
public function __construct(Configuration $config)
{
$this->config = $config;
if (!is_dir($this->config->upload_dir)) {
mkdir($this->config->upload_dir, 0755, true);
}
}
public function getAvailableDumps(): array
{
$files = [];
$iterator = new FilesystemIterator($this->config->upload_dir);
foreach ($iterator as $fileinfo) {
if (preg_match('/\.(sql|gz|csv)$/i', $fileinfo->getFilename())) {
$files[] = [
'name' => $fileinfo->getFilename(),
'size' => $fileinfo->getSize(),
'date' => date("Y-m-d H:i:s", $fileinfo->getMTime()),
'type' => $this->getFileType($fileinfo->getFilename()),
];
}
}
return $files;
}
private function getFileType(string $filename): string
{
if (preg_match('/\.sql$/i', $filename)) {
return 'SQL';
}
if (preg_match('/\.gz$/i', $filename)) {
return 'GZip';
}
if (preg_match('/\.csv$/i', $filename)) {
return 'CSV';
}
return 'Misc';
}
public function handleUpload(array $file): string
{
if ($file['error'] !== UPLOAD_ERR_OK) {
throw new RuntimeException('Error uploading file.');
}
$filename = preg_replace("/[^_A-Za-z0-9-\.]/i", '', basename($file['name']));
$filepath = $this->config->upload_dir . '/' . $filename;
if (file_exists($filepath)) {
throw new RuntimeException("File $filename already exists!");
}
if (!preg_match("/(\.(sql|gz|csv))$/i", $filename)) {
throw new RuntimeException('You may only upload .sql, .gz, or .csv files.');
}
if (!move_uploaded_file($file['tmp_name'], $filepath)) {
throw new RuntimeException("Error moving uploaded file to $filepath");
}
return $filename;
}
public function deleteFile(string $filename): void
{
$filename = preg_replace("/[^_A-Za-z0-9-\.]/i", '', basename($filename));
$filepath = $this->config->upload_dir . '/' . $filename;
if (!file_exists($filepath) || !is_file($filepath)) {
throw new RuntimeException("File $filename not found.");
}
// Basic security check to prevent deleting files outside of the upload directory
if (dirname(realpath($filepath)) !== realpath($this->config->upload_dir)) {
throw new RuntimeException("Invalid file path.");
}
if (!unlink($filepath)) {
throw new RuntimeException("Can't remove $filename.");
}
}
public function openFile(string $filename, bool &$gzipmode)
{
$filepath = $this->config->upload_dir . '/' . $filename;
if (!file_exists($filepath) || !is_file($filepath)) {
throw new RuntimeException("File $filename not found.");
}
if (dirname(realpath($filepath)) !== realpath($this->config->upload_dir)) {
throw new RuntimeException("Invalid file path.");
}
$gzipmode = preg_match("/\.gz$/i", $filename);
if ((!$gzipmode && !$file = @fopen($filepath, "r")) || ($gzipmode && !$file = @gzopen($filepath, "r"))) {
throw new RuntimeException("Can't open $filename for import.");
}
return $file;
}
}

18
templates/import.php Normal file
View File

@@ -0,0 +1,18 @@
<?php
// templates/import.php
?>
<div id="import-page-marker" data-delay="<?php echo htmlspecialchars($delaypersession, ENT_QUOTES, 'UTF-8'); ?>">
<p class="centr">Processing file: <b><?php echo htmlspecialchars($filename, ENT_QUOTES, 'UTF-8'); ?></b></p>
<p class="smlcentr">Starting from line: <span id="linenumber"><?php echo $start_line; ?></span></p>
<div id="progress">
<!-- progress table will be injected here by javascript -->
</div>
<div id="import_messages">
</div>
<p class="centr">
<a href="?" id="stop_button" class="button">STOP</a>
</p>
</div>

29
templates/layout.php Normal file
View File

@@ -0,0 +1,29 @@
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<title>BigDump: Staggered MySQL Dump Importer</title>
<meta http-equiv="CONTENT-TYPE" content="text/html; charset=iso-8859-1"/>
<meta http-equiv="CONTENT-LANGUAGE" content="EN"/>
<meta http-equiv="Cache-Control" content="no-cache"/>
<meta http-equiv="Pragma" content="no-cache"/>
<meta http-equiv="Expires" content="-1"/>
<meta name="robots" content="noindex, nofollow">
<link href="assets/style.css" rel="stylesheet">
</head>
<body>
<center>
<table width="780" cellspacing="0" cellpadding="0">
<tr><td class="transparent">
<div class="skin1">
<h1>BigDump: Staggered MySQL Dump Importer</h1>
</div>
<?php echo $content; ?>
<p class="centr">&copy; 2003-2024 <a href="mailto:alexey@ozerov.de">Alexey Ozerov</a> & contributors</p>
</td></tr>
</table>
</center>
<script src="assets/script.js"></script>
</body>
</html>

51
templates/start.php Normal file
View File

@@ -0,0 +1,51 @@
<?php
// templates/start.php
?>
<?php if (!empty($error)): ?>
<p class="error"><?php echo htmlspecialchars($error, ENT_QUOTES, 'UTF-8'); ?></p>
<?php endif; ?>
<?php if (!empty($success)): ?>
<p class="success"><?php echo htmlspecialchars($success, ENT_QUOTES, 'UTF-8'); ?></p>
<?php endif; ?>
<table width="100%" cellspacing="2" cellpadding="2">
<tr>
<th>Filename</th>
<th>Size</th>
<th>Date&amp;Time</th>
<th>Type</th>
<th>&nbsp;</th>
<th>&nbsp;</th>
</tr>
<?php foreach ($files as $file): ?>
<tr>
<td><?php echo htmlspecialchars($file['name'], ENT_QUOTES, 'UTF-8'); ?></td>
<td class="right"><?php echo htmlspecialchars($file['size'], ENT_QUOTES, 'UTF-8'); ?></td>
<td><?php echo htmlspecialchars($file['date'], ENT_QUOTES, 'UTF-8'); ?></td>
<td><?php echo htmlspecialchars($file['type'], ENT_QUOTES, 'UTF-8'); ?></td>
<td>
<a href="?action=import&file=<?php echo urlencode($file['name']); ?>">Start Import</a>
</td>
<td>
<a href="?action=delete&file=<?php echo urlencode($file['name']); ?>" onclick="return confirm('Are you sure you want to delete this file?');">Delete file</a>
</td>
</tr>
<?php endforeach; ?>
</table>
<?php if (empty($files)): ?>
<p>No uploaded SQL, GZ or CSV files found in the working directory</p>
<?php endif; ?>
<h3>Upload a dump file</h3>
<form method="POST" action="?action=upload" enctype="multipart/form-data">
<p>Dump file: <input type="file" name="dumpfile" size="60"></p>
<p><input type="submit" name="uploadbutton" value="Upload"></p>
</form>
<?php if ($db_connection_charset): ?>
<p>Note: The current mySQL connection charset is <i><?php echo htmlspecialchars($db_connection_charset, ENT_QUOTES, 'UTF-8'); ?></i>. Your dump file must be encoded in <i><?php echo htmlspecialchars($db_connection_charset, ENT_QUOTES, 'UTF-8'); ?></i> in order to avoid problems with non-latin characters.</p>
<?php endif; ?>

View File

@@ -0,0 +1,20 @@
<?php
use BigDump\Configuration;
use PHPUnit\Framework\TestCase;
class ConfigurationTest extends TestCase
{
public function testDefaultConfigIsLoaded()
{
$config = new Configuration();
$this->assertEquals('localhost', $config->db_server);
$this->assertEquals(3000, $config->linespersession);
}
public function testUploadDirIsCorrect()
{
$config = new Configuration();
$this->assertStringEndsWith('/data', $config->upload_dir);
}
}

50
tests/FileHandlerTest.php Normal file
View File

@@ -0,0 +1,50 @@
<?php
use BigDump\Configuration;
use BigDump\FileHandler;
use PHPUnit\Framework\TestCase;
class FileHandlerTest extends TestCase
{
private $config;
private $fileHandler;
private $testFile;
private $uploadDir;
protected function setUp(): void
{
$this->config = new Configuration();
$this->fileHandler = new FileHandler($this->config);
$this->uploadDir = $this->config->upload_dir;
$this->testFile = $this->uploadDir . '/test.sql';
if (!is_dir($this->uploadDir)) {
mkdir($this->uploadDir, 0777, true);
}
file_put_contents($this->testFile, 'test data');
}
protected function tearDown(): void
{
if (file_exists($this->testFile)) {
unlink($this->testFile);
}
// Small cleanup: remove data dir if empty
if (is_dir($this->uploadDir) && count(scandir($this->uploadDir)) == 2) {
rmdir($this->uploadDir);
}
}
public function testGetAvailableDumps()
{
$dumps = $this->fileHandler->getAvailableDumps();
$this->assertCount(1, $dumps);
$this->assertEquals('test.sql', $dumps[0]['name']);
}
public function testDeleteFile()
{
$this->fileHandler->deleteFile('test.sql');
$this->assertFileDoesNotExist($this->testFile);
}
}