Data modeling with excel

Excel Data Model (Table of Contents)

• Introduction to Data Model in Excel
• How to Create a Data Model in Excel?

Introduction to Data Model in Excel

The data model feature of Excel enables the easy building of relationships between easy reporting and their background data sets. It makes data analysis much easier. It allows the integration of data from a plethora of tables spread across multiple worksheets by simply building relationships between matching columns. It works completely behind the scene and greatly simplifies reporting features such as PivotTable etc.

Our article shall attempt to show how to create a pivot table from two tables by employing the Data Model feature, thus establishing a relationship between two table objects and thereby creating a Pivot Table.

How to Create a Data Model in Excel?

Let’s understand how to create the Data Model in Excel with a few examples.

You can download this Data Model Excel Template here – Data Model Excel Template

Example #1

• We have a list of products, and we have a shelving code for each product. We need a table where we have the shelving description along with the shelving codes. So how do we incorporate the shelving descriptions against each shelving code? Perhaps many of us would resort to using VLOOKUP here, but we shall altogether remove the need to use VLOOKUP here using Excel Data Model.

• The table on the left is the data table, and the table on the right is the lookup table. As we can see from the data, it is possible to create a relationship based on common columns.

• Now, the Data Model is only compatible with table objects. So, it might be necessary sometimes to convert data sets to table objects. To do that, follow the below steps.

1. Left-click anywhere in the data set.
2. Click the Insert tab and navigate to Table in the Tables group or simply press Ctrl+T.
3. Uncheck or check the My Table has the Header option. In our example, it does indeed have a header. Click OK.
4. While still focused on the new table, we need to provide a name that is meaningful in the Name box (towards the left of the formula bar).

In our example, we have named the table Personnel.

• Now we need to do the same process for the lookup table as well and name it Shelf Code.

Creating a Relationship

So firstly, we shall go to the Data tab and then select Relationships in the Data Tools subgroup. After we click on the Relationships option, in the beginning, since there is no relationship, hence we will have nothing.

We will first click on New to create a relationship. We will now need to provide the primary and the lookup table names from the drop-down list and then also mention the column which is common between the two tables so that we can establish the relationship between the two tables from the drop-down list of columns.

• Now, the primary table is the table that has the data. It is the primary data table – Table5. On the other hand, the Related table is the table that has the lookup data – it is our lookup table ShelfCodesTable. The primary table is the one that is analyzed based on the lookup table, which contains lookup data that will make the reported data, in the end, more meaningful.

• So, the common column between the two tables is the Shelf Code column. This is what we have used to establish the relationship between the two tables. Coming to the columns, the Column (foreign) is the one that refers to the data table where there can be duplicate values. On the other hand, the Related Column (primary) refers to the column in the lookup table where we have unique values. We are simply setting up the field to lookup values from the lookup table in the data table.
• Once we set this up, Excel would create a relationship between the two behind the scene. It integrates the data and creates a data model based on the common column. This is light on the memory requirements and much faster than using VLOOKUP in large workbooks. After defining the Data model, Excel would be treating these objects as Data Model tables instead of a worksheet table.
• Now to see what Excel has been up to, we can click on Manage Data Models in Data -> Data Tools.

• We can also get the diagrammatic representation of the data model by changing the view. We will click on the View option. This will open up the view options. We will then select the Diagram View. Then we will see the diagrammatic representation, showing the two tables and the relationship between them, i.e. the common column – Shelf Code.

• The diagram above shows a one-to-many relationship between the unique lookup table values and the data table with duplicated values.
• Now we will have to create a pivot table. To do that, we will go to the Insert tab and then click on the Pivot Table option.

In the Pivot table’s Create Pivot Table dialogue box, we will select the source as “Use this workbook’s Data Model”.

• This will create the Pivot table, and we can see that both the source tables are available in the source section.

• Now we shall create a pivot table showing the count of each person who has shelved items.

• We will select Personnel in the Rows section from Table 5 (data table), followed by Description (lookup table).

• Now we shall drag the Shelf Code from Table 5 into the Values section.

• Now we shall add Months from Table 5 to the Rows section.

• Or we could add the months as a filter and add it to the Filters section.

Example #2

• We now have Mr Basu running a factory called Basu Corporation. Mr Basu is trying to estimate the revenue for 2019 based on the data from 2018.
• We have a table where we have the revenue for 2018 and the subsequent revenue at different incremental levels.

• So, we have the revenue for 2018 – $1.5 M, and the minimum growth expected the following year is 12%. Mr Basu wants a table that will show the revenue at different incremental levels.
• We will create the following table for the projections at different incremental levels for 2019.

• Now we shall give the first Revenue row a reference to the estimated minimum revenue for 2019, i.e. $1.68 M.

• After using the formula, the answer is shown below.

• Now we shall select the entire table, i.e. D2:E12 and then go to Data ->Forecast ->What-If Analysis ->Data Table.

• This will open up the Data Table dialog box. Here we shall enter the minimum increment percent from cell B4 in the Column Input cell. The reason for that is that our projected estimated Growth percentages in the table are arranged in a columnar fashion.

• Once we click OK, the What-If Analysis will automatically populate the table with projected revenue at the different incremental percentages.

Example #3

• Now suppose we have the same scenario as above, except that now we also have another axis to consider. Suppose, in addition to showing the projected revenue in 2019 based on the data of 2018 and the minimum expected growth rate, we now also have the estimated discount rate.

• First, we shall have a table shown below.

• Now we shall give reference to the minimum projected revenue for 2019, i.e. cell B5 to cell D8.

• Now we shall select the entire table, i.e. D8:J18 and then go to Data ->Forecast ->What-If Analysis ->Data Table.

• This will open up the Data Table dialog box. Here we shall enter the minimum increment percent from cell B3 in the Column Input cell. The reason for that is that our projected estimated Growth percentages in the table are arranged in a columnar fashion. We shall now also additionally enter the minimum discount percent from cell B4 in the Row Input cell. The reason for that is that our projected discount percentages in the table are arranged in a row-wise fashion.

• Click OK. This will make the What-If Analysis automatically populate the table with projected revenue at the different incremental percentages as per the discount percentages.

Things to Remember About Data Model in Excel

• Upon successfully calculating the values from the data table, a simple Undo, i.e. Ctrl+Z, will not work. It is, however, possible to manually delete the values from the table.
• It is not possible to delete a single cell from the table. It is described as an array internally in Excel; hence we will have to delete all the values.
• We need to properly select the Row Input Cell and the Column Input Cell.
• The data table, unlike the Pivot Table, doesn’t need to be refreshed every time.
• Using the Data Model in Excel, we can improve performance and go easy on memory requirements in large worksheets.
• Data Models also makes our analysis much simpler as compared to using a number of complicated formulae all across the workbook.

Recommended Articles

This is a guide to Data Model in Excel. Here we discuss how to create Data Model in Excel along with practical examples and a downloadable excel template. You can also go through our other suggested articles –

1. Pivot Table Slicer
2. PowerPivot in Excel
3. Pivot Table Formula in Excel
4. Excel Pivot Table

Create a memory-efficient Data Model using Excel and the Power Pivot add-in

In Excel 2013 or later, you can create data models containing millions of rows, and then perform powerful data analysis against these models. Data models can be created with or without the Power Pivot add-in to support any number of PivotTables, charts, and Power View visualizations in the same workbook.

Note: This article describes data models in Excel 2013. However, the same data modeling and Power Pivot features introduced in Excel 2013 also apply to Excel 2016. There’s effectively little difference between these versions of Excel.

Although you can easily build huge data models in Excel, there are several reasons not to. First, large models that contain multitudes of tables and columns are overkill for most analyses, and make for a cumbersome Field List. Second, large models use up valuable memory, negatively affecting other applications and reports that share the same system resources. Finally, in Microsoft 365, both SharePoint Online and Excel Web App limit the size of an Excel file to 10 MB. For workbook data models that contain millions of rows, you’ll run into the 10 MB limit pretty quickly. See Data Model specification and limits.

In this article, you’ll learn how to build a tightly constructed model that’s easier to work with and uses less memory. Taking the time to learn best practices in efficient model design will pay off down the road for any model you create and use, whether you’re viewing it in Excel 2013, Microsoft 365 SharePoint Online, on an Office Web Apps Server, or in SharePoint 2013.

Consider also running the Workbook Size Optimizer. It analyzes your Excel workbook and if possible, compresses it further. Download the Workbook Size Optimizer.

In this article

Compression ratios and the in-memory analytics engine

Data models in Excel use the in-memory analytics engine to store data in memory. The engine implements powerful compression techniques to reduce storage requirements, shrinking a result set until it is a fraction of its original size.

On average, you can expect a data model to be 7 to 10 times smaller than the same data at its point of origin. For example, if you’re importing 7 MB of data from a SQL Server database, the data model in Excel could easily be 1 MB or less. The degree of compression actually achieved depends primarily on the number of unique values in each column. The more unique values, the more memory is required to store them.

Why are we talking about compression and unique values? Because building an efficient model that minimizes memory usage is all about compression maximization, and the easiest way to do that is to get rid of any columns that you don’t really need, especially if those columns include a large number of unique values.

Note: The differences in storage requirements for individual columns can be huge. In some cases, it is better to have multiple columns with a low number of unique values rather than one column with a high number of unique values. The section on Datetime optimizations covers this technique in detail.

Nothing beats a non-existent column for low memory usage

The most memory-efficient column is the one that you never imported in the first place. If you want to build an efficient model, look at each column and ask yourself whether it contributes to the analysis you want to perform. If it doesn’t or you aren’t sure, leave it out. You can always add new columns later if you need them.

Two examples of columns that should always be excluded

The first example relates to data that originates from a data warehouse. In a data warehouse, it’s common to find artifacts of ETL processes that load and refresh data in the warehouse. Columns like “create date”, “update date”, and “ETL run” are created when the data is loaded. None of these columns are needed in the model and should be deselected when you import data.

The second example involves omitting the primary key column when importing a fact table.

Many tables, including fact tables, have primary keys. For most tables, such as those that contain customer, employee, or sales data, you’ll want the table’s primary key so that you can use it to create relationships in the model.

Fact tables are different. In a fact table, the primary key is used to uniquely identify each row. While necessary for normalization purposes, it’s less useful in a data model where you want only those columns used for analysis or to establish table relationships. For this reason, when importing from a fact table, do not include its primary key. Primary keys in a fact table consume enormous amounts of space in the model, yet provide no benefit, as they cannot be used to create relationships.

Note: In data warehouses and multidimensional databases, large tables consisting of mostly numeric data are often referred to as “fact tables”. Fact tables typically include business performance or transaction data, such as sales and cost data points that are aggregated and aligned to organizational units, products, market segments, geographical regions, and so on. All of the columns in a fact table that contain business data or that can be used to cross-reference data stored in other tables should be included in the model to support data analysis. The column you want to exclude is the primary key column of the fact table, which consists of unique values that exist only in the fact table and nowhere else. Because fact tables are so huge, some of the biggest gains in model efficiency are derived from excluding rows or columns from fact tables.

How to exclude unnecessary columns

Efficient models contain only those columns that you’ll actually need in your workbook. If you want to control which columns are included in the model, you’ll have to use the Table Import Wizard in the Power Pivot add-in to import the data rather than the “Import Data” dialog box in Excel.

When you start the Table import Wizard, you select which tables to import.

For each table, you can click the Preview & Filter button and select the parts of the table that you really need. We recommend that you first uncheck all columns, and then proceed to check the columns you do want, after considering whether they are required for the analysis.

What about filtering just the necessary rows?

Many tables in corporate databases and data warehouses contain historical data accumulated over long periods of time. Additionally, you might find that the tables you’re interested in contain information for areas of the business not required for your specific analysis.

Using the Table Import wizard, you can filter out historical or unrelated data, and thus save a lot of space in the model. In the following image, a date filter is used to retrieve only rows that contain data for the current year, excluding historical data that won’t be needed.

What if we need the column; can we still reduce its space cost?

There are a few additional techniques you can apply to make a column a better candidate for compression. Remember that the only characteristic of the column that affects compression is the number of unique values. In this section, you’ll learn how some columns can be modified to reduce the number of unique values.

Modifying Datetime columns

In many cases, Datetime columns take a lot of space. Fortunately, there are a number of ways to reduce the storage requirements for this data type. The techniques will vary depend on how you use the column, and your comfort level in building SQL queries.

Datetime columns include a date part and a time. When you ask yourself whether you need a column, ask the same question multiple times for a Datetime column:

Do I need the time part?

Do I need the time part at the level of hours? , minutes? , Seconds? , milliseconds?

Do I have multiple Datetime columns because I want to calculate the difference between them, or just to aggregate the data by year, month, quarter, and so on.

How you answer each of these questions determines your options for dealing with the Datetime column.

All of these solutions require modification of a SQL query. To make query modification easier, you should filter out at least one column in every table. By filtering out a column, you change query construction from an abbreviated format (SELECT *) to a SELECT statement that includes fully-qualified column names, which are far easier to modify.

Let’s take a look at the queries that are created for you. From the Table Properties dialog box, you can switch to the Query editor and see the current SQL query for each table.

From Table Properties, select Query Editor.

The Query Editor shows the SQL query used to populate the table. If you filtered out any column during import, your query includes fully-qualified column names:

In contrast, if you imported a table in its entirety, without unchecking any column or applying any filter, you’ll see the query as “Select * from ”, which will be more difficult to modify:

Modifying the SQL query

Now that you know how to find the query, you can modify it to further reduce the size of your model.

For columns containing currency or decimal data, if you don’t need the decimals, use this syntax to get rid of the decimals:

If you need the cents but not fractions of cents, replace the 0 by 2. If you use negative numbers, you can round to units, tens, hundreds etc.

If you have a Datetime column named dbo.Bigtable.[Date Time] and you do not need the Time part, use the syntax to get rid of the time:

“SELECT CAST (dbo.Bigtable.[Date time] as date) AS [Date time]) “

If you have a Datetime column named dbo.Bigtable.[Date Time] and you need both the Date and Time parts, use multiple columns in the SQL query instead of the single Datetime column:

“SELECT CAST (dbo.Bigtable.[Date Time] as date ) AS [Date Time],

datepart(hh, dbo.Bigtable.[Date Time]) as [Date Time Hours],

datepart(mi, dbo.Bigtable.[Date Time]) as [Date Time Minutes],

datepart(ss, dbo.Bigtable.[Date Time]) as [Date Time Seconds],

datepart(ms, dbo.Bigtable.[Date Time]) as [Date Time Milliseconds]”

Use as many columns as you need to store each part in separate columns.

If you need hours and minutes, and you prefer them together as one time column, you can use the syntax :

Timefromparts(datepart(hh, dbo.Bigtable.[Date Time]), datepart(mm, dbo.Bigtable.[Date Time])) as [Date Time HourMinute]

If you have two datetime columns, like [Start Time] and [End Time], and what you really need is the time difference between them in seconds as a column called [Duration], remove both columns from the list and add:

“datediff(ss,[Start Date],[End Date]) as [Duration]”

If you use the keyword ms instead of ss, you’ll get the duration in milliseconds

Using DAX calculated measures instead of columns

If you’ve worked with the DAX expression language before, you might already know that calculated columns are used to derive new columns based on some other column in the model, while calculated measures are defined once in the model, but evaluated only when used in a PivotTable or other report.

One memory-saving technique is to replace regular or calculated columns with calculated measures. The classic example is Unit Price, Quantity, and Total. If you have all three, you can save space by maintaining just two and calculating the third one using DAX.

Which 2 columns you should keep?

In the above example, keep Quantity and Unit Price. These two have fewer values than the Total. To calculate Total, add a calculated measure like:

“TotalSales:=sumx(‘Sales Table’,’Sales Table’[Unit Price]*’Sales Table’[Quantity])”

Calculated columns are like regular columns in that both take up space in the model. In contrast, calculated measures are calculated on the fly and do not take space.

Conclusion

In this article, we talked about several approaches that can help you build a more memory-efficient model. The way to reduce the file size and memory requirements of a data model is to reduce the overall number of columns and rows, and the number of unique values appearing in each column. Here are some techniques we covered:

Removing columns is of course the best way to save space. Decide which columns you really need.

Sometimes you can remove a column and replace it with a calculated measure in the table.

You may not need all the rows in a table. You can filter out rows in the Table Import Wizard.

In general, breaking apart a single column into multiple distinct parts is a good way to reduce the number of unique values in a column. Each one of the parts will have a small number of unique values, and the combined total will be smaller than the original unified column.

In many cases, you also need the distinct parts to use as slicers in your reports. When appropriate, you can create hierarchies from parts like Hours, Minutes, and Seconds.

Many times, columns contain more information than you need them too. For example, suppose a column stores decimals, but you’ve applied formatting to hide all the decimals. Rounding can be very effective in reducing the size of a numeric column.

Now that you’ve done what you can to reduce the size of your workbook, consider also running the Workbook Size Optimizer. It analyzes your Excel workbook and if possible, compresses it further. Download the Workbook Size Optimizer.

Источник

Tutorial: Import Data into Excel, and Create a Data Model

Abstract: This is the first tutorial in a series designed to get you acquainted and comfortable using Excel and its built-in data mash-up and analysis features. These tutorials build and refine an Excel workbook from scratch, build a data model, then create amazing interactive reports using Power View. The tutorials are designed to demonstrate Microsoft Business Intelligence features and capabilities in Excel, PivotTables, Power Pivot, and Power View.

Note: This article describes data models in Excel 2013. However, the same data modeling and Power Pivot features introduced in Excel 2013 also apply to Excel 2016.

In these tutorials you learn how to import and explore data in Excel, build and refine a data model using Power Pivot, and create interactive reports with Power View that you can publish, protect, and share.

The tutorials in this series are the following:

Import Data into Excel 2013, and Create a Data Model

In this tutorial, you start with a blank Excel workbook.

The sections in this tutorial are the following:

At the end of this tutorial is a quiz you can take to test your learning.

This tutorial series uses data describing Olympic Medals, hosting countries, and various Olympic sporting events. We suggest you go through each tutorial in order. Also, tutorials use Excel 2013 with Power Pivot enabled. For more information on Excel 2013, click here. For guidance on enabling Power Pivot, click here.

Import data from a database

We start this tutorial with a blank workbook. The goal in this section is to connect to an external data source, and import that data into Excel for further analysis.

Let’s start by downloading some data from the Internet. The data describes Olympic Medals, and is a Microsoft Access database.

Click the following links to download files we use during this tutorial series. Download each of the four files to a location that’s easily accessible, such as Downloads or My Documents, or to a new folder you create:
> OlympicMedals.accdb Access database
> OlympicSports.xlsx Excel workbook
> Population.xlsx Excel workbook
> DiscImage_table.xlsx Excel workbook

In Excel 2013, open a blank workbook.

Click DATA > Get External Data > From Access. The ribbon adjusts dynamically based on the width of your workbook, so the commands on your ribbon may look slightly different from the following screens. The first screen shows the ribbon when a workbook is wide, the second image shows a workbook that has been resized to take up only a portion of the screen.

Select the OlympicMedals.accdb file you downloaded and click Open. The following Select Table window appears, displaying the tables found in the database. Tables in a database are similar to worksheets or tables in Excel. Check the Enable selection of multiple tables box, and select all the tables. Then click OK.

The Import Data window appears.

Note: Notice the checkbox at the bottom of the window that allows you to Add this data to the Data Model, shown in the following screen. A Data Model is created automatically when you import or work with two or more tables simultaneously. A Data Model integrates the tables, enabling extensive analysis using PivotTables, Power Pivot, and Power View. When you import tables from a database, the existing database relationships between those tables is used to create the Data Model in Excel. The Data Model is transparent in Excel, but you can view and modify it directly using the Power Pivot add-in. The Data Model is discussed in more detail later in this tutorial.

Select the PivotTable Report option, which imports the tables into Excel and prepares a PivotTable for analyzing the imported tables, and click OK.

Once the data is imported, a PivotTable is created using the imported tables.

With the data imported into Excel, and the Data Model automatically created, you’re ready to explore the data.

Explore data using a PivotTable

Exploring imported data is easy using a PivotTable. In a PivotTable, you drag fields (similar to columns in Excel) from tables (like the tables you just imported from the Access database) into different areas of the PivotTable to adjust how it presents your data. A PivotTable has four areas: FILTERS, COLUMNS, ROWS, and VALUES.

It might take some experimenting to determine which area a field should be dragged to. You can drag as many or few fields from your tables as you like, until the PivotTable presents your data how you want to see it. Feel free to explore by dragging fields into different areas of the PivotTable; the underlying data is not affected when you arrange fields in a PivotTable.

Let’s explore the Olympic Medals data in the PivotTable, starting with Olympic medalists organized by discipline, medal type, and the athlete’s country or region.

In PivotTable Fields, expand the Medals table by clicking the arrow beside it. Find the NOC_CountryRegion field in the expanded Medals table, and drag it to the COLUMNS area. NOC stands for National Olympic Committees, which is the organizational unit for a country or region.

Next, from the Disciplines table, drag Discipline to the ROWS area.

Let’s filter Disciplines to display only five sports: Archery, Diving, Fencing, Figure Skating, and Speed Skating. You can do this from within the PivotTable Fields area, or from the Row Labels filter in the PivotTable itself.

Click anywhere in the PivotTable to ensure the Excel PivotTable is selected. In the PivotTable Fields list, where the Disciplines table is expanded, hover over its Discipline field and a dropdown arrow appears to the right of the field. Click the dropdown, click (Select All)to remove all selections, then scroll down and select Archery, Diving, Fencing, Figure Skating, and Speed Skating. Click OK.

Or, in the Row Labels section of the PivotTable, click the dropdown next to Row Labels in the PivotTable, click (Select All) to remove all selections, then scroll down and select Archery, Diving, Fencing, Figure Skating, and Speed Skating. Click OK.

In PivotTable Fields, from the Medals table, drag Medal to the VALUES area. Since Values must be numeric, Excel automatically changes Medal to Count of Medal.

From the Medals table, select Medal again and drag it into the FILTERS area.

Let’s filter the PivotTable to display only those countries or regions with more than 90 total medals. Here’s how.

In the PivotTable, click the dropdown to the right of Column Labels.

Select Value Filters and select Greater Than….

Type 90 in the last field (on the right). Click OK.

Your PivotTable looks like the following screen.

With little effort, you now have a basic PivotTable that includes fields from three different tables. What made this task so simple were the pre-existing relationships among the tables. Because table relationships existed in the source database, and because you imported all the tables in a single operation, Excel could recreate those table relationships in its Data Model.

But what if your data originates from different sources, or is imported at a later time? Typically, you can create relationships with new data based on matching columns. In the next step, you import additional tables, and learn how to create new relationships.

Import data from a spreadsheet

Now let’s import data from another source, this time from an existing workbook, then specify the relationships between our existing data and the new data. Relationships let you analyze collections of data in Excel, and create interesting and immersive visualizations from the data you import.

Let’s start by creating a blank worksheet, then import data from an Excel workbook.

Insert a new Excel worksheet, and name it Sports.

Browse to the folder that contains the downloaded sample data files, and open OlympicSports.xlsx.

Select and copy the data in Sheet1. If you select a cell with data, such as cell A1, you can press Ctrl + A to select all adjacent data. Close the OlympicSports.xlsx workbook.

On the Sports worksheet, place your cursor in cell A1 and paste the data.

With the data still highlighted, press Ctrl + T to format the data as a table. You can also format the data as a table from the ribbon by selecting HOME > Format as Table. Since the data has headers, select My table has headers in the Create Table window that appears, as shown here.

Formatting the data as a table has many advantages. You can assign a name to a table, which makes it easy to identify. You can also establish relationships between tables, enabling exploration and analysis in PivotTables, Power Pivot, and Power View.

Name the table. In TABLE TOOLS > DESIGN > Properties, locate the Table Name field and type Sports. The workbook looks like the following screen.

Save the workbook.

Import data using copy and paste

Now that we’ve imported data from an Excel workbook, let’s import data from a table we find on a web page, or any other source from which we can copy and paste into Excel. In the following steps, you add the Olympic host cities from a table.

Insert a new Excel worksheet, and name it Hosts.

Select and copy the following table, including the table headers.

Источник