Python Combine Multiple Csv Files


Let’s now use the os.chdir (‘.’) to go up one working directory before saving our data: os.chdir ('.' ) pwd '/Users/jamesphoenix/Desktop/ImranAndJamesProjects/PythonForSEO/2bulkcsvoperations'. Combinedcsvdata.tocsv ('combinedcsvdata.csv') #Saving our combined csv data as a new file! Steps By Step to Merge Two CSV Files Step 1: Import the Necessary Libraries import pandas as pd Here all things are done using pandas python library. Step 2: Load the Dataset I have created two CSV datasets on Stocks Data one is a set of stocks and the other is the. Step 3: Merge the.

As this course is being progressively released, whenever a new article and video is released, after initially git cloning the repository. You will need to run this command within your command line / terminal (from the root directory of the course):

This will pull any recent changes that have been made on the version of the course and will allow you to easily get fresh content as it is added.

Learning Outcomes


Python Combine Two Csv Files

Python Combine Multiple Csv Files
  • To learn what the pd.concat() method is and how it works
  • Learn how to combine multiple csv files using Pandas

Firstly let’s say that we have 5, 10 or 100 .csv files. Combining all of these by hand can be incredibly tiring and definitely deserves to be automated. Therefore in today’s exercise, we’ll combine multiple csv files within only 8 lines of code.

For this tutorial, I’ve already prepared 5 top pages .csv reports from Ahrefs which can be found in the following directory:

One of the problems with automatically detecting csv files is that the names are dynamically generated. Therefore we will be using the .csv file extension name and a python package called glob to automatically detect all of the files ending with a .csv name within a specific working directory.

Python Combine Multiple Csv Files Into Sas

Import packages and set the working directory

You will need to change “/directory” to your specific directory.

By writing pwd within the command line, we can identify the exact file path that these Ahrefs top page .csv files are located in:

Let’s now move into our desired working directory where the csv files are:

Now let’s running !ls and !pwd just to show that we have changed directory:

Pro-tip: using ! before a linux command allows you to run the unix/linux commands within a jupyter notebook file!

Step 2: Use Global To Match The Pattern ‘.csv’

We will now match the file pattern (‘.csv’) within all of the files located in the current working directory.

Step 3: Let’s Combine All Of The Files Within The List And Export as a CSV

In the code below we will read all of the csv’s and will then use the pd.concat() method to stack every dataframe one on top of another.

But before we do that, let’s make sure that we can get one result within a pandas dataframe by adding the appropriate encoding:

  • UTF-16 (This is a specific encoding type).
  • t (tab delimited data).

Now let’s break down what the above line of code does, firstly we loop over all of the filenames and assign them one by one to the f variable. Each csv file is then read & converted into a pandas dataframe with:

Then we concatenate all of the dataframes together and stack them one on top of each other using:

That’s it, within 8 lines of code you’re now able to easily combine as many .csv files as you want!

  • Remember that all of the csv files must have the same columns otherwise you will not be able to effectively concatenate them!

Python Combine Multiple Csv Files Excel

Step 4 Save Your New DataFrame To CSV

Let’s now use the os.chdir(‘..’) to go up one working directory before saving our data: