Pandas excel file format cannot be determined you must specify an engine manually

I am not sure why I am getting this error although sometimes my code works fine!

Excel file format cannot be determined, you must specify an engine manually.

Here below is my code with steps:

1- list of columns of customers Id:

customer_id = ["ID","customer_id","consumer_number","cus_id","client_ID"]

2- The code to find all xlsx files in a folder and read them:

l = [] #use a list and concat later, faster than append in the loop
for f in glob.glob("./*.xlsx"):
    df = pd.read_excel(f).reindex(columns=customer_id).dropna(how='all', axis=1)
    df.columns = ["ID"] # to have only one column once concat
    l.append(df)
all_data  = pd.concat(l, ignore_index=True) # concat all data

I added the engine openpyxl

df = pd.read_excel(f, engine="openpyxl").reindex(columns = customer_id).dropna(how='all', axis=1)

Now I got a different error:

BadZipFile: File is not a zip file

pandas version: 1.3.0
python version: python3.9
os: MacOS

is there a better way to read all xlsx files from a folder ?

Источник

When attempting to access an Excel file in Python, if you receive the “Excel file format cannot be determined, you must specify an engine manually” message. This is because the Excel file format cannot be determined. So, you must manually define an engine. We’ll show you some ways to fix this error in the sections below. Scroll down and continue reading.

When you access an Excel file to work with, the operating system creates a temporary file that looks like this: ~$employees.xls.

Then, in our project, two .xlsx files with nearly identical names appear, and Python cannot determine which file needs to be read. This is the main cause of this error.

Consider the following example to get a better understanding.

When you access the Excel file, the system will generate a temporary file that looks like this:

import pandas as pd

# Read an Excel file using read_excel()
print(pd.read_excel('employees.xls'))

If you run the above code, you will receive the error:

Excel file format cannot be determined, you must specify an engine manually.

How to resolve this error?

Two ways to fix this error are saving the Excel file in a different format and manually specifying an engine. Below we will guide you to do that in detail.

Saving the Excel file in a different format

You can save an Excel worksheet as another file using the Save As command.

Navigate to File > Save As > Choose Browse.

In the Save As dialogue box, select the file format for the worksheet under Save as type, such as CSV (Comma delimited). As shown below:

After saving successfully, you will see the following csv file:

Instead of reading the Excel file with read_excel(), we will now read the CSV file with read_csv(). Like this:

import pandas as pd

# Read a CSV file using read_csv()
print(pd.read_csv('employees.csv'))

Output:

   ID    Name  Gender
0   1   David    Male
1   2   Lucas    Male
2   3   Betty  Female
3   4  Rachel  Female

As you can see from the output, we can read the contents of the original Excel file without any errors.

Specifying the engine manually

Another way to fix this error is specifying the engine manually when you open the Excel file. This tells Python which parser to use and should help it correctly identify the file’s format.

We can use two main packages in this case: xlrd and openpyxl. We recommend using openpyxl instead of xlrd because it is more versatile and easier to use.

To install openpyxl, run the following command in your terminal:

pip install openpyxl

After installing openpyxl, you should be able to read your Excel file without any issues. Let’s re-run our faulty code to make sure!

This time, we also opened the Excel file to force the operating system to create a temporary file. Like this:

Then try executing the following code again:

import pandas as pd

# Read an Excel file using read_excel()
print(pd.read_excel('employees.xls'))

Output:

   ID    Name  Gender
0   1   David    Male
1   2   Lucas    Male
2   3   Betty  Female
3   4  Rachel  Female

As you can see, with the openpyxl module installed, we can work with the Excel file without converting it to another format.

Summary

The leading cause for the error “Excel file format cannot be determined, you must specify an engine manually” is that Microsoft software creates temporary files automatically. Installing the openpyxl module is the simplest and quickest way to fix it. After reading this article, we are sure you will not encounter this error again. Share this article with your friends if you found it helpful.

Lopez

Hi, I’m Cora Lopez. I have a passion for teaching programming languages such as Python, Java, Php, Javascript … I’m creating the free python course online. I hope this helps you in your learning journey.

Name of the university: HCMUE
Major: IT
Programming Languages: HTML/CSS/Javascript, PHP/sql/laravel, Python, Java

Источник

Solution 1

Found it. When an excel file is opened for example by MS excel a hidden temporary file is created in the same directory:

~$datasheet.xlsx

So, when I run the code to read all the files from the folder it gives me the error:

Excel file format cannot be determined, you must specify an engine manually.

When all files are closed and no hidden temporary files ~$filename.xlsx in the same directory the code works perfectly.

Solution 2

Also make sure you’re using the correct pd.read_* method. I ran into this error when attempting to open a .csv file with read_excel() instead of read_csv(). I found this handy snippet here to automatically select the correct method by Excel file type.

if file_extension == 'xlsx':
    df = pd.read_excel(file.read(), engine='openpyxl')
elif file_extension == 'xls':
    df = pd.read_excel(file.read())
elif file_extension == 'csv':
    df = pd.read_csv(file.read())

Solution 3

https://stackoverflow.com/a/32241271/17411729

link to an answer on how to remove hidden files

Mac = go to folder press cmd + shift + .
will show the hidden file, delete it, run it back.

Solution 4

In macOS, an «invisible file» named «.DS_Store» is automatically generated in each folder. For me, this was the source of the issue. I solved the problem with an if statement to bypass the «invisible file» (which is not an xlsx, so thus would trigger the error)

for file in os.scandir(test_folder):
    filename = os.fsdecode(file)
    if '.DS_Store' not in filename:
        execute_function(file)

Solution 5

Looks like an easy fix for this one. Go to your excel file, whether it is xls or xlsx or any other extension, and do «save as» from file icon. When prompted with options. Save it as CSV UTF-8(Comma delimited)(*.csv)

Comments

I am not sure why I am getting this error although sometimes my code works fine!

Excel file format cannot be determined, you must specify an engine manually.

Here below is my code with steps:

1- list of columns of customers Id:
```
customer_id = ["ID","customer_id","consumer_number","cus_id","client_ID"]
```
2- The the code to find all xlsx files in a folder and read them:
```
l = [] #use a list and concat later, faster than append in the loop
for f in glob.glob("./*.xlsx"):
    df = pd.read_excel(f).reindex(columns=customer_id).dropna(how='all', axis=1)
    df.columns = ["ID"] # to have only one column once concat
    l.append(df)
all_data  = pd.concat(l, ignore_index=True) # concat all data
```
I added the engine openpyxl

df = pd.read_excel(f, engine="openpyxl").reindex(columns = customer_id).dropna(how='all', axis=1)

Now I got a different error:
```
BadZipFile: File is not a zip file
```
pandas version: 1.3.0
python version: python3.9
os: MacOS

is there a better way to read all xlsx files from a folder ?
Thank you for pointing out a potentially duplicated question. However, there are two things that you may consider: 1) to mention it only as a comment to the question, rather than an answer 2) If the solution in the SO page that you referred is not exactly the same, you should include the steps that you took too, not only the link
Thanks for making me aware, I will try to keep that in mind the next time would love to move my answer to the comments, unfortunately, I’m not allowed to make comments until I get 50 reps.

Recents

Источник

have a 1 GB excel sheet with xls format (old excel), and I can’t read it with pandas

df = pd.read_excel("filelocation/filename.xls",engine = "xlrd")

XLRDError: Unsupported format, or corrupt file: Expected BOF record; found b'<html>rn'

and if removed the engine it sends this error

ValueError: Excel file format cannot be determined, you must specify an engine manually

any advice will be appreciated thanks

>Solution :

One of these options should work:

data = pandas.read_table(r"filelocation/filename.xls")

data = pandas.read_html("filelocation/filename.xls")

Otherwise, try another HTML parse, I agree with @AKX, this doesn’t look like an excel file.

Источник

Issue

I am not sure why I am getting this error although sometimes my code works fine!

Excel file format cannot be determined, you must specify an engine manually.

Here below is my code with steps:

1- list of columns of customers Id:

customer_id = ["ID","customer_id","consumer_number","cus_id","client_ID"]

2- The code to find all xlsx files in a folder and read them:

l = [] #use a list and concat later, faster than append in the loop
for f in glob.glob("./*.xlsx"):
    df = pd.read_excel(f).reindex(columns=customer_id).dropna(how='all', axis=1)
    df.columns = ["ID"] # to have only one column once concat
    l.append(df)
all_data  = pd.concat(l, ignore_index=True) # concat all data

I added the engine openpyxl

df = pd.read_excel(f, engine="openpyxl").reindex(columns = customer_id).dropna(how='all', axis=1)

Now I got a different error:

BadZipFile: File is not a zip file

pandas version: 1.3.0
python version: python3.9
os: MacOS

is there a better way to read all xlsx files from a folder ?

Solution

Found it. When an excel file is opened for example by MS excel a hidden temporary file is created in the same directory:

~$datasheet.xlsx

So, when I run the code to read all the files from the folder it gives me the error:

Excel file format cannot be determined, you must specify an engine manually.

When all files are closed and no hidden temporary files ~$filename.xlsx in the same directory the code works perfectly.

Answered By – Mtaly

This Answer collected from stackoverflow, is licensed under cc by-sa 2.5 , cc by-sa 3.0 and cc by-sa 4.0

Источник

The error is thrown when reading a non-excel file. Let’s have a look at deeply

Error code:

import pandas as pd 

excel = pd.read_excel('/content/Book1.pdf') 
print(excel)

pd.read_excel method use pd.ExcelFile class to read excel files in the current directory. It takes path and engine as an argument.

path: It can be one of a file-like object, xlrd workbook or openpyxl workbook. If a string or path object, expected to be a path to a .xls, .xlsx, .xlsb, .xlsm, .odf, .ods, or .odt file.

engine: Each file format has own special engine like below.

— xlrd supports old-style Excel files (.xls).

— openpyxl supports newer Excel file formats.

— odf supports OpenDocument file formats (.odf, .ods, .odt).

— pyxlsb supports Binary Excel files.

So, if our path is not one of above mentioned file, The code will give an error.

Fix code:

import pandas as pd 

excel = pd.read_excel('/content/Book1.xlsx') 
print(excel)

Источник

Содержание

Pandas excel file format cannot be determined you must specify an engine manually
Excel file format cannot be determined, you must specify an engine manually #
Your operating system creating temporary lock files #
Make sure you have the openpyxl module installed #
Using pandas.read_excel vs pandas.read_csv #
Checking for a file’s extension before reading it #
ValueError when reading excel from sharepoint to python
2 Answers 2
Solution to fix the “Excel file format cannot be determined, you must specify an engine manually” error in Python
What causes the “Excel file format cannot be determined, you must specify an engine manually” error?
How to resolve this error?
Saving the Excel file in a different format
Specifying the engine manually
Summary
[Solved] Excel file format cannot be determined, you must specify an engine manually
How Excel file format cannot be determined, you must specify an engine manually Error Occurs ?
How To Solve Excel file format cannot be determined, you must specify an engine manually Error ?
Solution 1: define engine
Solution 2: Close Opened File
Summary
ENH: The XLS_SIGNATURE is too restrictive #41225
Comments

Reading time В· 3 min

Excel file format cannot be determined, you must specify an engine manually #

The «ValueError: Excel file format cannot be determined, you must specify an engine manually» error occurs for multiple reasons:

Your operating system creates temporary lock files named

$file_name.xlsx .

You tried to use the pd.read_excel() method to read a csv file or the pd.read_csv() method to read an xlsx file.

Your files have mislabeled extensions.

Your operating system creating temporary lock files #

The most common cause of the error is that your operating system creates temporary lock files named

If you have the excel files open in an application, close the application and adjust your code to ignore these files.

The code sample assumes that you have an xlsx file located in the same directory as your Python script, e.g. example.xlsx .

Here is the output of running the python main.py command.

We used the in operator to ignore files that contain

$ and read the other files with the xlsx extension.

Make sure you have the openpyxl module installed #

In order to read .xlsx files, you have to set the engine to openpyxl, so make sure you have the module installed.

The openpyxl library is used to read and write Excel 2010 xlsx , xlsm , xltx , xltm files.

Using pandas.read_excel vs pandas.read_csv #

Another common cause of the error is trying to read a .csv file using the pandas.read_excel method.

You should use the pandas.read_csv method when reading .csv files and the pandas.read_excel method when reading .xlsx and .xls files.

If you try to read a .csv file with the pandas.read_excel() method, the error is raised.

Similarly, if you try to read a .xlsx or .xls file with the pandas.read_csv method, the error is raised.

Checking for a file’s extension before reading it #

Here is an example of how to check for a file’s extension before reading it.

The file in the example has a .xlsx extension, so the openpyxl engine is used.

Источник

I am trying to read an excel file from sharepoint to python.

Q1: There are two URLs for the file. If I directly copy the link of the file, I get:

If I click into folders from the webpage one after another, until I click and open the excel file, the URL now is:

Which one should I use?

Q2: My code below:

It works until the pd.read_excel() , where I get ValueError.

I don’t know where it went wrong and if there will be further problems with loading. It will be highly appreciated if someone could warn me of the problems or leave an example.

2 Answers 2

If you take a look at the pandas documentation for ‘read_excel’ (https://pandas.pydata.org/docs/reference/api/pandas.read_excel.html), you’ll see that there is an ‘engine’ parameter.

Try the different options and see which one works, since your error is saying that an engine has to be specified manually.

If this is correct, in the future, take the error messages literally and check the documentation

I have tried different URLs (and how to obtain them), and received different binary files. They are either a line of code status (like 403) or warning, or something that looks like a header. So I believe the problem is the URL format.

Here (github.com/vgrem) I found the answer.

It basically says that for ClientContext you need an absolute URL,

And for File you need a relative path, but with overlap with the URL:

The RELATIVE_PATH can be found like this:

Go to the folder of the file in Teams (or on the webpage).

Choose the file, Open in app (Excel).

In Excel, File -> Property , copy the path and adapt to the above format.

Источник

Solution to fix the “Excel file format cannot be determined, you must specify an engine manually” error in Python

Table of Contents

What causes the “Excel file format cannot be determined, you must specify an engine manually” error?

When you access an Excel file to work with, the operating system creates a temporary file that looks like this:

Then, in our project, two .xlsx files with nearly identical names appear, and Python cannot determine which file needs to be read. This is the main cause of this error.

Consider the following example to get a better understanding.

When you access the Excel file, the system will generate a temporary file that looks like this:

If you run the above code, you will receive the error:

How to resolve this error?

Two ways to fix this error are saving the Excel file in a different format and manually specifying an engine. Below we will guide you to do that in detail.

Saving the Excel file in a different format

You can save an Excel worksheet as another file using the Save As command.

Navigate to File > Save As > Choose Browse.

In the Save As dialogue box, select the file format for the worksheet under Save as type, such as CSV (Comma delimited). As shown below:

After saving successfully, you will see the following csv file:

Instead of reading the Excel file with read_excel() , we will now read the CSV file with read_csv() . Like this:

Output:

As you can see from the output, we can read the contents of the original Excel file without any errors.

Specifying the engine manually

Another way to fix this error is specifying the engine manually when you open the Excel file. This tells Python which parser to use and should help it correctly identify the file’s format.

We can use two main packages in this case: xlrd and openpyxl. We recommend using openpyxl instead of xlrd because it is more versatile and easier to use.

To install openpyxl, run the following command in your terminal:

After installing openpyxl, you should be able to read your Excel file without any issues. Let’s re-run our faulty code to make sure!

This time, we also opened the Excel file to force the operating system to create a temporary file. Like this:

Then try executing the following code again:

Output:

As you can see, with the openpyxl module installed, we can work with the Excel file without converting it to another format.

Summary

Name of the university: HCMUE
Major: IT
Programming Languages: HTML/CSS/Javascript, PHP/sql/laravel, Python, Java

Источник

[Solved] Excel file format cannot be determined, you must specify an engine manually

Hello Guys, How are you all? Hope You all Are Fine. Today Whenever I am trying to open xls file I am facing following error some times. However Sometimes its Working fine Excel file format cannot be determined, you must specify an engine manually in python. So Here I am Explain to you all the possible solutions here.

Without wasting your time, Let’s start This Article to Solve This Error.

How Excel file format cannot be determined, you must specify an engine manually Error Occurs ?

Whenever I am trying to open xls file I am facing following error some times. However Sometimes its Working fine.

How To Solve Excel file format cannot be determined, you must specify an engine manually Error ?

How To Solve Excel file format cannot be determined, you must specify an engine manually Error?

To Solve Excel file format cannot be determined, you must specify an engine manually Error I am trying to open my file1 and my file1 is opened in MS excel So that whenever you are trying to open Already Opened File then you will face this error. To solve this error Just close opened file and now Try to open with your python code and your error will be solve.
Excel file format cannot be determined, you must specify an engine manually.

Solution 1: define engine

Just define engine Just like below.

Solution 2: Close Opened File

I am trying to open my file1 and my file1 is opened in MS excel So that whenever you are trying to open Already Opened File then you will face this error.

To solve this error Just close opened file and now Try to open with your python code and your error will be solve.

Summary

It’s all About this issue. Hope all solution helped you a lot. Comment below Your thoughts and your queries. Also, Comment below which solution worked for you?

Источник

ENH: The XLS_SIGNATURE is too restrictive #41225

Pandas 1.2.4 fails to open XLS files generated by Lotus 1-2-3 because they have a different header than the one expected in XLS_SIGNATURE ( io.excel._base line 1001 and then 1050).

I know I’m likely to be the only person in the world with this issue, but my boss still uses Lotus 1-2-3 🙄

I’m willing to submit a pull request with my fix if people are okay with it. The fix basically involves changing XLS_SIGNATURE to a list:

I don’t know if the second XLS_SIGNATURES value is «good,» but it works. I can attach an XLS that was exported by 1-2-3 if it would be beneficial to others, or if they’d be able to help me find a better second value for XLS_SIGNATURES . I tried looking through the byte-code for anything similar to the «DOCFILE» XLS signature, but didn’t see anything.

The text was updated successfully, but these errors were encountered:

Thanks for the report! An example sheet would be helpful to further diagnose.

@rhshadrach See attached. If you have any suggestions I’d be happy to put together a PR.

For future reference: OpenOffice has made available the various BIFF files for testing.

Thanks @geoffrey-eisenbarth — documentation is hard to find, but I think this is a BIFF4 format:

This agrees with your code: b»x09x04x06x00x00x00″ (note I’m replacing your usage of t ). What you suggested would be a welcome PR.

However, it seems likely to me there are other file signatures that will still go unrecognized. One of the shortfalls of the current pandas implementation is that if the user specifies engine but we can’t verify the file is correct, we will raise without just trying the engine.

Instead, I think inspect_excel_format should return None if a format cannot be determined instead of raising. Then, we should raise if engine is None with a message to the effect «Excel file format cannot be determined, you must specify an engine manually», and otherwise attempt to use the engine the user specified.

Would you be interested in putting up a PR for this @geoffrey-eisenbarth?

@rhshadrach I agree with your suggested route, and I’d love to submit the PR. Hoping to have time tomorrow to dig more into the code and reference your suggestion with inspect_excel_format . Should be a fast PR.

Источник

Pandas excel file format cannot be determined you must specify an engine manually

How to resolve this error?

Saving the Excel file in a different format

Specifying the engine manually

Summary

Solution 1

Solution 2

Solution 3

Solution 4

Solution 5

Related videos on Youtube

Comments

Recents

Related

>Solution :

Issue

Solution