Pacejet

Troubleshooting Data Quality Issues When Using Bulk Import Export

Overview

The Pacejet Bulk Import Export utility provides useful features to automate common setup tasks as you implement Pacejet for the first time or as you activate new features in the future. Using the Bulk Import Export tool, administrators can easily export, update, and import Classifications, Commodities, Packages, Products, and many other types of common data. This article outlines the most common data quality challenges administrators will encounter while using the Bulk Import Export utility with suggestions for how to correct data for easier imports.


Basics of Bulk Import Export with Data Maps

A core feature of the Bulk Import Export utility is a data map creation utility to drag and drop fields into a file format, save the format under a name, and then use this data map to export or import various types of Pacejet data. Flexible data maps allow the Pacejet import and export process to adapt to file formats produced or consumed by other systems, whether imports are one-time or recurring. Users have fine-grained control over the specific data fields and ordering of fields in file formats.


Templates as a Way to Quick Start the Process

One of the challenges of providing very flexible control over the import export process is the complexity of creating data maps from scratch. The data mapping utility, and really any data mapping utility, can be tedious to use in crafting templates with 20, 40, 60, or more fields. To help out, Pacejet also provides a set of common templates with predefined maps for various types of Pacejet data. Starting with a template, users can modify the fields, ordering, or other characteristics to meet specific needs.


What Could Possibly Go Wrong?

Do a quick google search on "data quality issues for new system implementations" and it's easy to see that data management is one of the most challenging tasks for IT system setup and ongoing management. Tasks like loading up a list of the packages used by the shipping team seems straightforward but there are plenty of data quality-related issues which can derail this simple activity. Some of the most common data quality issues faced by analysts and organizations include:

  • Duplicates = Multiple copies of the same records can take a huge toll on system storage, skew results in reporting, and cause unpredictable processing results.

  • Inconsistent Formats = Multiple variations of data formats, including mixed use of capitalization, punctuation, country codes, decimal precision can all create many issues and problems.

  • Incomplete Data = When data is not entered completely or correctly, certain files can easily become corrupted and become useless (e.g. a missing or corrupt zip or postal code).

  • Invalid Lengths or Content = Data with NULL or blank values, use of special characters, very large or oversize values, and other invalid content types can cause processing to fail, reports to work improperly, calculations to fail, and other problems.


The Most Common Data Quality Issues

While the best advice for reaching data quality nirvana is to avoid duplicates, use consistent formats, prevent incomplete data, and block invalid lengths or invalid content types before attempting any data imports. Based on Pacejet's experience with many shippers, though, here are some of the most common data quality issues to focus on first.

  • Classifications = Avoid duplicate classifications where the same product code is used without a unique subcode (e.g. NMFC-12345 Rate 50 and NMFC-12345 Rate 55). Using the same code and subcode with a different description is ALSO considered a duplicate to be avoided. From a formatting perspective, make sure that Excel manipulation does NOT remove leading zeros from product codes and subcodes AND that your descriptions NEVER exceed 200 characters.

  • Commodities = Commodities with the same name but different descriptions are considered duplicates and cannot be used. Missing commodity codes, non-numeric harmonization codes, or descriptions longer than 200 characters are considered incomplete or invalid. And country code values MUST always follow the legal 2-digit ISO standard.

  • Packages = Package records are uniquely identified by the package number field, so duplicates with different dimensions and weight are NOT supported.

  • Products = Product records are uniquely identified by the product number field, so duplicates with different descriptions and other fields are NOT supported. Another key requirement is that product numbers can never exceed 30 characters and descriptions can be no longer than 200 characters.

  • Scan Codes = Duplicate scan codes can be a tricky issue since they are not always intuitive to some administrators. Scan codes MUST be unique across ALL products, not just within a product. And scan codes with different quantities must be unique across ALL products as well.