{"id":31371,"date":"2025-01-17T08:30:43","date_gmt":"2025-01-17T08:30:43","guid":{"rendered":"https:\/\/doc.probance.com\/?post_type=tutoriels&#038;p=31371"},"modified":"2025-01-17T10:46:35","modified_gmt":"2025-01-17T10:46:35","slug":"interpret-correct-and-recover-rejected-data-during-daily-imports","status":"publish","type":"tutoriels","link":"https:\/\/doc.probance.com\/en\/knowledge-base\/interpret-correct-and-recover-rejected-data-during-daily-imports\/","title":{"rendered":"Interpret, correct, and recover rejected data during daily imports"},"content":{"rendered":"\n<p>During daily processes and manual data imports, processed files are renamed by adding a <code>.ok<\/code> extension followed by the date and time the file was handled by your Probance project. Each of these files also generates several rejection files containing rows that could not be imported into the database.<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-style-note is-layout-flow wp-block-quote-is-layout-flow\">\n<p>If a file is not renamed on the SFTP after the daily processes, it indicates that it was not processed.<\/p>\n<\/blockquote>\n\n\n\n<p><strong>Event flows<\/strong> generate <strong>5 rejection files<\/strong>, while <strong>profile and catalog flows<\/strong> generate <strong>4 rejection files<\/strong>. An additional file is generated for flows sent via the real-time API.<\/p>\n\n\n\n<p>You can find these rejection files on the SFTP of your main account in the <code>upload\/rejects<\/code> directory. They are retained there for <strong>7 days<\/strong>.<\/p>\n\n\n\n<p>This directory also includes rejection files for flows from your partners, even if they have individual deposit directories for confidentiality purposes. However, your partners do not have access to the rejection files for the flows they generate.<\/p>\n\n\n\n<p>Reviewing these files will allow you to:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Understand the reason for the rejection.<\/li>\n\n\n\n<li>Plan data recovery after corrections using a Probance naming convention.<\/li>\n<\/ul>\n\n\n\n<div style=\"height:20px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<h2 class=\"wp-block-heading\">1. Types of Rejections<\/h2>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>loadingreject<\/strong><\/h4>\n\n\n\n<p>Returns all rows that do not comply with the file format defined in the flow specifications (e.g., delimiter, number of columns, end-of-line character, encapsulation).<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>checkschemareject<\/strong><\/h4>\n\n\n\n<p>Returns all rows where a required value is missing or a string exceeds the maximum length defined in the flow specifications.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>convertreject<\/strong><\/h4>\n\n\n\n<p>Rejects all rows with data types that do not match those defined in the flow specifications (e.g., integer, date pattern, decimal numbers).<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>filterrowreject<\/strong><\/h4>\n\n\n\n<p>If you have defined filters on your fields in Cockpit, this rejection type includes rows where a column does not comply with its filter.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>lookupreject<\/strong><\/h4>\n\n\n\n<p>This type of rejection applies to event flows. It includes rows corresponding to events for which the customer (client identifier) is unknown in the customer table.<\/p>\n\n\n\n<p>Lookup rejections may have two causes:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>The <code>id_user<\/code> line was never sent in the profile flow.<\/li>\n\n\n\n<li>The <code>id_user<\/code> line was sent in the profile flow but was rejected during the import.<\/li>\n<\/ol>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Jsonreject<\/strong><\/h4>\n\n\n\n<p>This rejection type indicates the nature of import errors encountered when processing lines received via JSON calls.<\/p>\n\n\n\n<div style=\"height:20px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<h2 class=\"wp-block-heading\">2. Recovering Rejected or Missing Data<\/h2>\n\n\n\n<p>PHM natively supports concatenating your CSV files by adding an extension to your file names.<\/p>\n\n\n\n<p>For example, for the file <code>clients_prospects_DDMMYYYY.csv<\/code>, you can upload the following files:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><code>clients_prospects_DDMMYYYY.csv<\/code><\/li>\n\n\n\n<li><code>clients_prospects_DDMMYYYY-00.csv<\/code><\/li>\n\n\n\n<li><code>clients_prospects_DDMMYYYY-01.csv<\/code><\/li>\n\n\n\n<li><code>clients_prospects_DDMMYYYY-02.csv<\/code><\/li>\n\n\n\n<li>&#8230;up to <code>clients_prospects_DDMMYYYY-99.csv<\/code><\/li>\n<\/ul>\n\n\n\n<p>These files will be processed in order, from <code>-00<\/code> to <code>-99<\/code>, ending with the daily file without an extension.<\/p>\n\n\n\n<p>If a row appears multiple times in these recovery files (for profile data and not events), the value in the row of the last file will be retained during data import.<\/p>\n\n\n\n<p>You can view information about the expected file naming for each flow in your Probance One interface under <strong>Administration \/ Technical<\/strong>.<\/p>\n<div id=\"wpmem_restricted_msg\"><p>This content is restricted to site members.  If you are an existing user, please log in.  New users may register below.<\/p><\/div><div id=\"wpmem_login\"><a id=\"login\"><\/a><form action=\"https:\/\/doc.probance.com\/en\/knowledge-base\/interpret-correct-and-recover-rejected-data-during-daily-imports\/\" method=\"POST\" id=\"wpmem_login_form\" class=\"form\"><input type=\"hidden\" id=\"_wpmem_login_nonce\" name=\"_wpmem_login_nonce\" value=\"f834b2436e\" \/><input type=\"hidden\" name=\"_wp_http_referer\" value=\"\/en\/wp-json\/wp\/v2\/tutoriels\/31371\" \/><fieldset><legend>Existing Users Log In<\/legend><label for=\"log\">Username or Email<\/label><div class=\"div_text\"><input name=\"log\" type=\"text\" id=\"log\" value=\"\" class=\"username\" required  \/><\/div><label for=\"pwd\">Password<\/label><div class=\"div_text\"><input name=\"pwd\" type=\"password\" id=\"pwd\" class=\"password\" required  \/><\/div><input name=\"a\" type=\"hidden\" value=\"login\" \/><input name=\"redirect_to\" type=\"hidden\" value=\"https:\/\/doc.probance.com\/en\/knowledge-base\/interpret-correct-and-recover-rejected-data-during-daily-imports\/\" \/><div class=\"button_div\"><input name=\"rememberme\" type=\"checkbox\" id=\"rememberme\" value=\"forever\" \/>&nbsp;<label for=\"rememberme\">Remember Me<\/label>&nbsp;&nbsp;<input type=\"submit\" name=\"Submit\" value=\"Log In\" class=\"buttons\" \/><\/div><\/fieldset><\/form><\/div><div id=\"wpmem_reg\"><a id=\"register\"><\/a><form name=\"form\" method=\"post\" action=\"https:\/\/doc.probance.com\/en\/knowledge-base\/interpret-correct-and-recover-rejected-data-during-daily-imports\/\" id=\"wpmem_register_form\" class=\"form\"><input type=\"hidden\" id=\"_wpmem_register_nonce\" name=\"_wpmem_register_nonce\" value=\"6cc3867c56\" \/><input type=\"hidden\" name=\"_wp_http_referer\" value=\"\/en\/wp-json\/wp\/v2\/tutoriels\/31371\" \/><fieldset><legend>New User Registration<\/legend><label for=\"username\" class=\"text\">Choose a Username<span class=\"req\">*<\/span><\/label><div class=\"div_text\"><input name=\"username\" type=\"text\" id=\"username\" value=\"\" class=\"textbox\" required  \/><\/div><label for=\"first_name\" class=\"text\">First Name<span class=\"req\">*<\/span><\/label><div class=\"div_text\"><input name=\"first_name\" type=\"text\" id=\"first_name\" value=\"\" class=\"textbox\" required  \/><\/div><label for=\"last_name\" class=\"text\">Last Name<span class=\"req\">*<\/span><\/label><div class=\"div_text\"><input name=\"last_name\" type=\"text\" id=\"last_name\" value=\"\" class=\"textbox\" required  \/><\/div><label for=\"user_email\" class=\"text\">Email<span class=\"req\">*<\/span><\/label><div class=\"div_text\"><input name=\"user_email\" type=\"email\" id=\"user_email\" value=\"\" class=\"textbox\" required  \/><\/div><input name=\"a\" type=\"hidden\" value=\"register\" \/><input name=\"wpmem_reg_page\" type=\"hidden\" value=\"https:\/\/doc.probance.com\/en\/knowledge-base\/interpret-correct-and-recover-rejected-data-during-daily-imports\/\" \/><div class=\"button_div\"><input name=\"submit\" type=\"submit\" value=\"Register\" class=\"buttons\" \/><\/div><div class=\"req-text\"><span class=\"req\">*<\/span>Required field<\/div><\/fieldset><\/form><\/div>","protected":false},"excerpt":{"rendered":"<p>During daily processes and manual data imports, processed files are renamed by adding a .ok extension followed by the date and time the file was handled by your Probance project. Each of these files also generates several rejection files containing rows that could not be imported into the database. If a file is not renamed [&hellip;]<\/p>\n","protected":false},"author":6,"featured_media":0,"template":"","sujets":[788],"class_list":["post-31371","tutoriels","type-tutoriels","status-publish","hentry","sujets-data-processing"],"_links":{"self":[{"href":"https:\/\/doc.probance.com\/en\/wp-json\/wp\/v2\/tutoriels\/31371","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/doc.probance.com\/en\/wp-json\/wp\/v2\/tutoriels"}],"about":[{"href":"https:\/\/doc.probance.com\/en\/wp-json\/wp\/v2\/types\/tutoriels"}],"author":[{"embeddable":true,"href":"https:\/\/doc.probance.com\/en\/wp-json\/wp\/v2\/users\/6"}],"wp:attachment":[{"href":"https:\/\/doc.probance.com\/en\/wp-json\/wp\/v2\/media?parent=31371"}],"wp:term":[{"taxonomy":"sujets","embeddable":true,"href":"https:\/\/doc.probance.com\/en\/wp-json\/wp\/v2\/sujets?post=31371"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}