Перевести excel в mysql

О преобразовании XLS в SQL запросы

Конвертер Excel в SQL была создана для быстрого преобразования Excel файлов в запросы SQL(Structured Query Language) для вставки в базу данных. Этот удивительный инструмент поддерживает настраиваемое имя таблицы и несколько команд MySQL для вставки. Нет необходимости загружать или устанавливать какое-либо программное обеспечение.

В результате вы получите несколько запросов:

• SQL-запрос для создания таблицы. Но это всего лишь пример «create table»; на самом деле не рекомендуется выполнять этот запрос, потому что он не содержит индексы, ключи, правильные типы и т.д.
• Insert или Replace запросы.

Как это работает?

Просто выберите нужные параметры и загрузите Excel(XLS или XLSX) файл и вы сразу получите SQL запросы.

После преобразования вы можете выполнить эти команды на вашем SQL сервере и добавить все данные в вашу базу данных.

Microsoft Excel has been a traditional choice as a spreadsheet application for organizations across the world. The ease of access, power formulas, and the ability to make visually stunning reports has made Microsoft Excel is widely used tool. However, managing thousands of sheets can be a time-consuming and resource-intensive task that may also lead to some errors. A good solution is to import your Excel files to a scalable, reliable, and fast relational database management system like MySQL.

You can Import Excel into MySQL either by using simple SQL commands like LOAD DATA or opt for third-party tools. By importing your Excel files to MySQL, you can leverage the powerful MySQL engine to quickly query data and perform complex data manipulations.

In this article, you will learn how to effectively Import Excel into MySQL using 4 different methods.

What is Microsoft Excel?

Microsoft Excel is a spreadsheet application developed by Microsoft to organize and record data in a tabular format. Launched in 1985, Microsoft Excel is a widely used Spreadsheet software in businesses worldwide allowing you to organize and manipulate data through formulas in columns and rows. MS Excel is loaded with functionalities to perform several operations such as calculations, pivot tables, graph tools, etc.

Compatible with Microsoft Windows, macOS, Android, and iOS, MS Excel is included with the MS Office Suite. It is typically used by firms to record, organize and analyze financial data. MS Excel also offers VBA programming that can assist you in automating repetitive tasks.

Key Features of Microsoft Excel

• Conditional Formatting: MS Excel allows you to apply formats to a cell or range of cells, and have that formatting change depending on the value of the cell or the value of a formula.
• Built-in Formulae: MS Excel provides you with basic and advanced built-in formulae for sum, average, minimum, etc. You can use formula auditing to graph or trace the relationship between cells and formulas with blue arrows. You can track precedents (cells that provide data for a particular cell) or the cells that depend on the value of a particular cell. 
• Data Protection: Excel spreadsheets can be password protected on your laptop or PC. MS Excel also provides a robust system to recover data.
• Data Sorting & Filtering: You can easily perform the basic data sorting and filtering operation on your data in excel. MS Excel also provides Advanced filtering options for complex criteria.

What is MySQL?

MySQL is a popular Open-Source Relational Database Management System. MySQL data is stored in tables consisting of rows and columns. Introduced in 1995, MySQL is completely developed, distributed, and maintained by Oracle Corporation. Looking over the technical side, MySQL is written in the C and C ++ languages.

MySQL is compatible with multiple platforms such as Microsoft Windows, Oracle Solaris, AIX, Symbian, Linux, and macOS. You can use SQL commands to retrieve, change or add data from your MySQL tables. MySQL is also a vital component of the Modern LAMP stack, which consists of a Linux-based Operating System, Apache Web Server, MySQL Database, and PHP for processing.

SQL can also be used to define the table schema or even configure the relationship between the tables in the database. Client-Side Graphical User Interfaces such as MySQL WorkBench, SequelPro, or DBVisualizer can be used to type the SQL commands and the server will respond with the requested information. MySQL also offers a paid Enterprise version which includes premium support services and several extensions that can be installed as Server Plugins.

Key Features of MySQL

MySQL is widely used by organizations like Facebook, Flickr, Twitter, Wikipedia, and YouTube. This is due to the following set of MySQL’s eye-catching features: 

• Ease of Operation: With MySQL, you can work with a diverse set of programming languages ​​such as PHP, PERL, C, C ++, and JAVA. It also provides a set of intuitive tools to simplify time-consuming tasks such as Server Management, Reporting, and Data Analysis.
• Support for Several Data Structures: Allowing you to work with a broader range of datasets, MySQL offers full support for several Data structures like JSON and Geospatial data, as well as Logical, Numeric, Alphanumeric, Date, and Time data types.
• First-rate Performance: MySQL delivers best-in-class query performance across a variety of clustered servers. It provides a fast-loading utility with a separate memory cache and table index partitioning. MySQL can also efficiently handle fluctuating workloads and process large volumes of data at optimal speeds. This allows you to store more than 50 million rows of data in a table with a table’s default file size limit of 4 GB which can be increased to a theoretical limit of 8 million terabytes (TB).
• Open-Source: MySQL is under the GNU General Public License (GPL), which means it will always be freely available. It provides you the freedom to customize it on Oracle’s Open-Source MySQL codebase based on your needs.
• Localization: To make it user-friendly, MySQL supports multiple character sets, including latin1 (cp1252), german, big5, ujis, and more. You can also set the error message language that the server provides to the client.
• Data Security: MySQL provides full control over the Data Access Control settings. Robust mechanisms such as Access Privilege Systems and User Account Management allow you to regulate who can view or use your MySQL data. MySQL sets the highest standard with Host-based Verification and Password Encryption.

Why should you import Excel into MySQL?

Microsoft Excel is an easy-to-use spreadsheet application that allows you to directly access data and perform a combination of operations on this data. As you collect data and store it in your separate excel files, it becomes a huge task to track each sheet with millions of records in it. To simplify this, you can import Excel into MySQL. MySQL is a scalable, fast, and reliable database management system that allows you to store and query millions of rows of data using simple SQL commands.

Using MYSQL, you can also connect your Excel tables with one another tables using primary and foreign keys. MySQL can easily handle large datasets and allows you to manipulate data, filter data, update data and combine it. You can import Excel into MySQL and enjoy better speed, accuracy, & consistency for your ever-growing data needs.

• Method 1: Using Hevo Data to import Excel into MySQL
• Method 2: Using Sqlizer.io to Import Excel into MySQL
• Method 3: Using the Load Data Statement to Import Excel into MySQL
• Method 4: Import Excel into MySQL using phpMyAdmin

Method 1: Using Hevo Data to import Excel into MySQL

Hevo Data is a No-code Data Pipeline solution that can help you move data from 150+ data sources (including 40+ free sources like Google Drive & Google Sheets) to your desired destination such as MySQL, Data Warehouses, or BI tools in a completely hassle-free & automated manner. Using Hevo you can easily upload your Excel files present on your Google Drive to your MySQL Database.

Hevo also supports MySQL as a Source for loading data to a destination of your choice. Hevo is fully managed and completely automates the process of not only loading data from 150+ data sources but also enriching the data and transforming it into an analysis-ready form without having to write a single line of code. Its fault-tolerant architecture ensures that the data is handled in a secure, consistent manner with zero data loss.

Hevo Data takes care of all your data preprocessing needs and lets you focus on key business activities and draw a more powerful insight on how to generate more leads, retain customers, and take your business to new heights of profitability. It provides a consistent & reliable solution to manage data in real-time and always have analysis-ready data in your desired destination. 

Take a look at some of the salient features of Hevo:

• Blazing-fast Setup: Straightforward interface for new customers to work on, with minimal setup time.
• Reliability at Scale – With Hevo, you get a world-class fault-tolerant architecture that scales with zero data loss and low latency. 
• Monitoring and Observability – Monitor pipeline health with intuitive dashboards that reveal every stat of pipeline and data flow. Bring real-time visibility into your ELT with Alerts and Activity Logs 
• Stay in Total Control – When automation isn’t enough, Hevo offers flexibility – data ingestion modes, ingestion, and load frequency, JSON parsing, destination workbench, custom schema management, and much more – for you to have total control.    
• Transparent Pricing – Say goodbye to complex and hidden pricing models. Hevo’s Transparent Pricing brings complete visibility to your ELT spend. Choose a plan based on your business needs. Stay in control with spend alerts and configurable credit limits for unforeseen spikes in data flow. 
• Auto-Schema Management – Correcting improper schema after the data is loaded into your warehouse is challenging. Hevo automatically maps source schema with destination warehouse so that you don’t face the pain of schema errors.
• 24×7 Customer Support – With Hevo you get more than just a platform, you get a partner for your pipelines. Discover peace with round the clock “Live Chat” within the platform. What’s more, you get 24×7 support even during the 14-day full-feature free trial.

Sign up here for a 14-Day Free Trial!

Without the need for converting your .xlsx file to .csv, you effortlessly import Excel into MySQL using Hevo by following the simple steps given below:

• Step 1: Upload your Excel files into your Google Drive.
• Step 2: Login to Hevo and naviagte to Pipleines > + Create. Select Google Drive as your Source and configure your Drive account by selecting the authentication method for connecting to Google Drive.

• Step 3: Finally, provide a name for your Pipeline and select all the folders whose Excel files you want to import to MySQL. Then, click on the Continue button.

• Step 4: For completing the process to import Excel into MySQL, you can start by providing your MySQL database credentials such as your authorized Username and Password, along with information about your Host IP Address and Port Number value. You will also need to provide a name for your database and a unique name for this destination.

Method 2: Using Sqlizer.io to Import Excel into MySQL

To import Excel into MySQL, you can use Sqlizer.io to convert your Excel files into SQL commands. You can achieve this by following the simple instructions given below: 

• Step 1: Click on the Browse button and select the Excel file you want to import to MySQL.
• Step 2: Select MySQL as your desired database. According to your excel file, check or uncheck My File has a Header Row.
• Step 3: Based on your Excel file, check Use CHECK IF TABLE EXISTS. For using the active worksheet, check the Use the active worksheet box.
• Step 4: You can also choose to enter the Worksheet name. For this example, the worksheet name Students is mentioned.
• Step 5: To import Excel into MySQL, you can either provide the data cell range or check the Convert the whole worksheet box to import the whole data.
• Step 6: Finally, name the table in which to import excel into MySQL. Then, click on the Convert My File button.

• Step 7: After your file is converted, you will get the following message. Now, download the queries or you can even copy the queries to execute them on MySQL.

After the queries have been executed, you can view your data in MySQL using the SELECT command.

Method 3: Using the Load Data Statement to Import Excel into MySQL

Using the LOAD DATA Statement, you can import Excel data into MySQL. To do that, follow the simple steps given below:

• Step 1: You have to first convert your .xlsx file to .csv file. For that, open up the excel file that you need to import.
• Step 2: Navigate to File > Save As. Select CSV (Comma delimited)(*.csv) as your file type and click on the Save button. 

• Step 3: Now, enter the LOAD DATA command shown below in your MySQL Workbench import Excel into MySQL.

# MySQL Version 8.0.27
LOAD DATA INFILE
'C:/ProgramFiles/MySQLExcel/import excel into mysql.csv'
INTO TABLE new_file
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY 'n'
IGNORE 1 ROWS;  

In the above process to import an excel file in MySQL workbench, the .csv file data will be added to the new_file table in your MySQL database without the first row as it contains the row headers. For this example, the .csv file uses a comma as the delimiter to separate 2 fields. You can also use FIELDS TERMINATED BY ‘t’ or FIELDS TERMINATED BY ‘ ‘ if your file’s fields are separated using a tab or a single space respectively. 

Alternatively, you can use the LOAD DATA command directly from the Command Line (CMD). 

Method 4: Import Excel into MySQL using phpMyAdmin

You can also import excel into MySQL using phpMyAdmin by first converting your .xlsx files to .csv files. For that, you have to first download and phpMyAdmin tool on your system. For this example, a new table named tb_students_phpmyadmin has been created in the MySQL Database. To import excel into MySQL, you can follow the simple steps given below: 

• Step 1: In phpMyAdmin, click on the Import tab and choose your desired .csv file. 
• Step 2: Now, enter the format-specific options and once you are done, click on the Go button present in the bottom right corner of your screen.

After clicking on the Go button, you will observe the data import confirmation messages for all your rows. 

• Step 3: Using the SELECT statement, you can check if the process to import excel into MySQL was successful or not.

Conclusion

In this article, you have learned how to effectively import Excel into MySQL using 4 different methods. Using Method 2, you convert your Excel file rows into SQL queries via sqlizer.io which you, later on, execute on your MySQL Workbench or Command-Line. Though this can be a time-consuming process if you are dealing with excel files containing thousands of rows. You can also use the LOAD DATA statement or a database administration tool like phpMyAdmin, though these require you to first convert the Excel files into CSV files. 

These 3 methods are a good option if you only need to rarely import Excel into MySQL with no requirement for processes like Data Cleaning & Data Standardisation. However, when you need to frequently import excel files with millions of rows of data in real-time that require complex data transformation, then you can opt for a No-Code Automated Data Integration platform like Hevo Data! 

Visit our Website to Explore Hevo

Hevo Data is a No-code Data Pipeline solution that can help you import data from a vast sea of sources like Google Drive to your desired destination such as MySQL, Data Warehouses, or BI tools. Using Hevo you can easily upload your Excel files present on your Google Drive to your MySQL Database. Hevo also supports MySQL as a Source for loading data to a destination of your choice. It is a reliable, completely automated, and secure service that doesn’t require you to write any code!  

If you are using MySQL as your Database Management System and searching for a no-fuss alternative to manual data integration, then Hevo can effortlessly automate this for you. Hevo, with its strong integration with 150+ sources (Including 40+ Free Sources like Google Drive & Google Sheets), allows you to not only export & load data but also transform & enrich your data & make it analysis-ready in a jiffy.

Want to take Hevo for a ride? Sign Up for a 14-day free trial and simplify your Data Integration process. Do check out the pricing details to understand which plan fulfills all your business needs.

Tell us about your experience of completing the process to Import Excel into  MySQL! Share your thoughts with us in the comments section below.

Время на прочтение
6 мин

Количество просмотров 105K

Введение

Здравствуй, $habrauser!

Бывает так, что вам нужно импортировать файл Excel в базу MySQL, но готового решения нигде нет. Вот и я, когда меня попросил друг поискать легкий способ импорта, сперва решил

загуглить

поискать решение. Увы, запрос php excel to mysql не дал ничего конкретного, или же описанные способы были довольно таки не удобны. Тогда же я решил найти библиотеку для работы с Excel на PHP, и мне попалась PHPExcel. Но опять же меня ждало разочарование, запрос phpexcel to mysql не дал ничего путного (я ленивый пользователь и дальше 1й страницы не хожу). В итоге я решил создать свой

велосипед

скрипт, которым и хочу поделиться с вами.

Начало

Итак, библиотеку я нашел, скачал и начал разбираться. Для начала нужно было подключить библиотеку и создать подключение к базе, что совсем не сложно:

require_once "PHPExcel.php";

$connection = new mysqli("localhost", "user", "pass", "base");
$connection->set_charset("utf8");

Далее нужно открыть файл Excel для чтения:

$PHPExcel_file = PHPExcel_IOFactory::load("./file.xlsx");

После открытия файла, нам нужно перебрать все листы в нем и каждый добавить в базу MySQL (можно и 1 конкретный, но об этом позже):

foreach ($PHPExcel_file->getWorksheetIterator() as $worksheet) {
    // ...
}

Ну а теперь самое интересное…

Перебор и добавление

Мы будем исходить из того, что таблицы у нас нет (или есть, но с другими данными) и ее нужно создать. Для этого нам нужно получить имена для столбцов (в соответствии с просьбой друга, имена могут находиться в 1 строчке таблицы):

// Строка для названий столбцов таблицы MySQL
$columns_str = "";
// Количество столбцов на листе Excel
$columns_count = PHPExcel_Cell::columnIndexFromString($worksheet->getHighestColumn());

// Перебираем столбцы листа Excel и генерируем строку с именами через запятую
for ($column = 0; $column < $columns_count; $column++) {
    $columns_str .= ($columns_name_on1line ? "column" . $column : $worksheet->getCellByColumnAndRow($column, 1)->getCalculatedValue()) . ",";
}

// Обрезаем строку, убирая запятую в конце
$columns_str = substr($columns_str, 0, -1);

Далее удаляем таблицу из базы, если она существовала, и создаем новую:

$connection->query("DROP TABLE IF EXISTS exceltable");
$connection->query("CREATE TABLE exceltable (" . str_replace(",", " TEXT NOT NULL,", $columns_str) . " TEXT NOT NULL)");

Как видно из кода, значения будут иметь тип TEXT. Теперь приступаем собственно к перебору ячеек и добавления их в базу. Конечно, такой алгоритм не сложно найти на просторах Stack Overflow, однако было замечено, что происходить ошибка при попытки чтения объединенных ячеек (точнее несоответствие количества столбцов и значений в запросе). Это я и решил учесть:

// Количество строк на листе Excel
$rows_count = $worksheet->getHighestRow();

// Перебираем строки листа Excel
for ($row = 1; $row <= $rows_count; $row++) {
    // Строка со значениями всех столбцов в строке листа Excel
    $value_str = "";

    // Перебираем столбцы листа Excel
    for ($column = 0; $column < $columns_count; $column++) {
        // Строка со значением объединенных ячеек листа Excel
        $merged_value = "";
        // Ячейка листа Excel
        $cell = $worksheet->getCellByColumnAndRow($column, $row);

        // Перебираем массив объединенных ячеек листа Excel
        foreach ($worksheet->getMergeCells() as $mergedCells) {
            // Если текущая ячейка - объединенная,
            if ($cell->isInRange($mergedCells)) {
                // то вычисляем значение первой объединенной ячейки, и используем её в качестве значения
                // текущей ячейки
                $merged_value = $worksheet->getCell(explode(":", $mergedCells)[0])->getCalculatedValue();
                break;
            }
        }

        // Проверяем, что ячейка не объединенная: если нет, то берем ее значение, иначе значение первой
        // объединенной ячейки
        $value_str .= "'" . (strlen($merged_value) == 0 ? $cell->getCalculatedValue() : $merged_value) . "',";
    }

    // Обрезаем строку, убирая запятую в конце
    $value_str = substr($value_str, 0, -1);

    // Добавляем строку в таблицу MySQL
    $connection->query("INSERT INTO exceltable (" . $columns_str . ") VALUES (" . $value_str . ")");
}

Все дело в функцию!

Конечно, данный скрипт был бы гораздо удобнее, если бы все объединить в функцию. Поэтому итоговый результат получается такой:

// Подключаем библиотеку
require_once "PHPExcel.php";

// Функция преобразования листа Excel в таблицу MySQL, с учетом объединенных строк и столбцов.
// Значения берутся уже вычисленными. Параметры:
//     $worksheet - лист Excel
//     $connection - соединение с MySQL (mysqli)
//     $table_name - имя таблицы MySQL
//     $columns_name_line - строка с именами столбцов таблицы MySQL (0 - имена типа column + n)
function excel2mysql($worksheet, $connection, $table_name, $columns_name_line = 0) {
  // Проверяем соединение с MySQL
  if (!$connection->connect_error) {
    // Строка для названий столбцов таблицы MySQL
    $columns_str = "";
    // Количество столбцов на листе Excel
    $columns_count = PHPExcel_Cell::columnIndexFromString($worksheet->getHighestColumn());

    // Перебираем столбцы листа Excel и генерируем строку с именами через запятую
    for ($column = 0; $column < $columns_count; $column++) {
      $columns_str .= ($columns_name_line == 0 ? "column" . $column : $worksheet->getCellByColumnAndRow($column, $columns_name_line)->getCalculatedValue()) . ",";
    }

    // Обрезаем строку, убирая запятую в конце
    $columns_str = substr($columns_str, 0, -1);

    // Удаляем таблицу MySQL, если она существовала
    if ($connection->query("DROP TABLE IF EXISTS " . $table_name)) {
      // Создаем таблицу MySQL
      if ($connection->query("CREATE TABLE " . $table_name . " (" . str_replace(",", " TEXT NOT NULL,", $columns_str) . " TEXT NOT NULL)")) {
        // Количество строк на листе Excel
        $rows_count = $worksheet->getHighestRow();

        // Перебираем строки листа Excel
        for ($row = $columns_name_line + 1; $row <= $rows_count; $row++) {
          // Строка со значениями всех столбцов в строке листа Excel
          $value_str = "";

          // Перебираем столбцы листа Excel
          for ($column = 0; $column < $columns_count; $column++) {
            // Строка со значением объединенных ячеек листа Excel
            $merged_value = "";
            // Ячейка листа Excel
            $cell = $worksheet->getCellByColumnAndRow($column, $row);

            // Перебираем массив объединенных ячеек листа Excel
            foreach ($worksheet->getMergeCells() as $mergedCells) {
              // Если текущая ячейка - объединенная,
              if ($cell->isInRange($mergedCells)) {
                // то вычисляем значение первой объединенной ячейки, и используем её в качестве значения
                // текущей ячейки
                $merged_value = $worksheet->getCell(explode(":", $mergedCells)[0])->getCalculatedValue();
                break;
              }
            }

            // Проверяем, что ячейка не объединенная: если нет, то берем ее значение, иначе значение первой
            // объединенной ячейки
            $value_str .= "'" . (strlen($merged_value) == 0 ? $cell->getCalculatedValue() : $merged_value) . "',";
          }

          // Обрезаем строку, убирая запятую в конце
          $value_str = substr($value_str, 0, -1);

          // Добавляем строку в таблицу MySQL
          $connection->query("INSERT INTO " . $table_name . " (" . $columns_str . ") VALUES (" . $value_str . ")");
        }
      } else {
        return false;
      }
    } else {
      return false;
    }
  } else {
    return false;
  }

  return true;
}

// Соединение с базой MySQL
$connection = new mysqli("localhost", "user", "pass", "base");
// Выбираем кодировку UTF-8
$connection->set_charset("utf8");

// Загружаем файл Excel
$PHPExcel_file = PHPExcel_IOFactory::load("./file.xlsx");

// Преобразуем первый лист Excel в таблицу MySQL
$PHPExcel_file->setActiveSheetIndex(0);
echo excel2mysql($PHPExcel_file->getActiveSheet(), $connection, "excel2mysql0", 1) ? "OKn" : "FAILn";

// Перебираем все листы Excel и преобразуем в таблицу MySQL
foreach ($PHPExcel_file->getWorksheetIterator() as $index => $worksheet) {
  echo excel2mysql($worksheet, $connection, "excel2mysql" . ($index != 0 ? $index : ""), 1) ? "OKn" : "FAILn";
}

Заключение

Что ж, надеюсь данная статья поможет вам. Ну, или, если вы захотите

изобрести свой велосипед, но только с моторчиком

написать свой скрипт, эта статья поможет вам начать.

P.S.

Это моя первая, и думаю, не последняя статья. Поэтому жду ваших советов и поправок, как тут принято, в комментариях.

Update

Вижу, все-таки, мне удалось создать небольшую дискуссию, но не все понимают, почему было сделано именно так. Постараюсь объяснить.

Во-первых: с этим должен был работать пожилой человек, которому будет трудновато объяснить как сохранить файл в CSV, при этом не потеряв данные (а такое исключать нельзя, к тому же у них свой формат на файл XLS, который приходит сверху) и, тем более, как это импортировать через phpMyAdmin (который, кстати, с версии 3.4.5 не поддерживает XLS/XLSX, советую посмотреть почему) или подобное. Так что это не подходит.

Во-вторых: все это должно быть расположено на хостинге, и установка модулей как на сервер, так и для локальных программ не подходит (к тому же там Linux, а не Windows, как некоторые подумали).

В-третьих: это дело проводится раз в полгода, однако от безделья я решил написать такую функцию, способную обобщить импорт (вдруг кому нужно).

Теперь о хорошем: переписал данную функцию в класс, исправил кое-что и добавил возможность экспорта из MySQL в Excel. Забрать можно отсюда.

Извините, что не ответил в комментариях, решил что в самой статье будет уместние.

Conversion methods

                    java -jar client-0.0.5.jar convert --output-format=mysql file.xlsx output-dir/
                

curl -F files[]=@file.xlsx 'https://www.rebasedata.com/api/v1/convert?outputFormat=mysql&errorResponse=zip' -o output.zip

How long does it take?

The conversion process takes between 15 seconds and multiple minutes. It depends on the size of the database.

You can also use your favourite tool

• Read Excel using PHP
• Read Excel using Python
• Read Excel using Ubuntu

Why use RebaseData?

• Strong security.
• Convert online without installing Microsoft Excel.
• Works with Windows, Mac and Linux.
• Use an API to convert your databases.
• Professional support.

Terms

• We don’t guarantee for the success or correctness of the conversion
• You are only allowed to convert your own database files
• By using RebaseData, you agree to our general terms